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INPUT  DEVICE  CHARACTERISTICS  CONTRIBUTE  TO  PERFORMANCE  DURING 
TRAINING  TO  OPERATE  A  SIMULATED  MICRO-UNMANNED  AERIAL  VEHICLE 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Unmanned  aerial  vehicles  (UAVs)  have  demonstrated  usefulness  in  a  variety  of  military 
environments,  and  the  U.S.  Army  has  embraced  this  technology  (Cambone,  Krieg,  Pace,  & 
Wells,  2005;  Military.com,  2008).  Micro-unmanned  aerial  vehicles  (MAVs)  are  small,  man- 
portable,  UAVs  intended  for  use  at  the  small  unit  level.  As  this  technology  is  increasingly 
utilized,  systematic  training  requirements  and  performance  metrics  for  vehicle  operators  need  to 
be  addressed. 

Previously,  researchers  at  the  United  States  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  developed  a  research  environment  to  examine  training  issues  related 
to  manual  operator  control  of  MAVs  (Durlach,  Neumann,  &  Billings,  2008;  Billings  &  Durlach, 
2008).  This  research  platform  consisted  of  a  simulated  operator  control  unit  (OCU),  which 
provided  sensory  imagery  from  the  onboard  cameras  of  a  simulated  MAV,  altimeter  information, 
a  digital  map,  and  other  components  critical  to  the  successful  operation  of  the  MAV.  Operator 
training  missions  were  created,  and  various  performance  metrics  were  assessed  for  usefulness.  In 
the  previous  experiments,  performance  for  flight  skill  missions  was  measured  via  completion 
times,  and  performance  for  reconnaissance  missions  was  measured  via  the  number  of  targets 
identified.  It  was  found  that  the  mission  completion  measures  for  flight  skill  missions  were 
sensitive  to  human-interface  manipulations,  whereas  the  number  of  targets  identified  for 
reconnaissance  missions  was  not.  The  interface  manipulation  involved  whether  manual  flight 
was  controlled  by  a  game  controller  or  a  point  and  click  method  using  a  mouse.  Thus,  the 
specific  finding  was  that  flight  skill  missions  were  completed  more  quickly  with  the  game 
controller  than  with  the  mouse.  The  current  research  followed  up  these  findings  in  two  ways. 
First,  it  explored  the  sensitivity  of  temporal  measures  of  performance  for  simulated 
reconnaissance  missions.  Second,  it  explored  the  characteristics  of  the  game  controller  vs.  the 
mouse  configurations  that  may  contributed  to  performance  advantages  in  the  control  of  the 
simulated  MAV.  More  specifically,  this  research  investigated  three  characteristics  of  the  game 
controller  that  may  have  led  to  a  performance  advantage:  attention  (focused  vs.  divided),  control 
of  vehicle  speed  (user-controlled  vs.  system-controlled),  and  movement  ability  (one  direction  vs. 
multiple  directions  simultaneously). 

Procedure: 

Data  were  collected  from  50  participants  recruited  from  the  local  university.  Participants 
were  randomly  assigned  to  one  of  five  input  device  configurations,  through  which  we  were  able 
to  isolate  attention,  directional  movement,  and  control  of  speed  to  see  if  any  of  these  game 
controller  characteristics  contributed  to  performance  advantages.  Participants  sat  in  front  of  the 
OCU  to  complete  the  experiment.  The  layout  of  the  OCU  was  the  same  as  it  was  for  our  prior 
experiment,  but  the  methods  of  input  were  different.  In  addition,  the  general  procedure  was  the 
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same  as  in  our  prior  research  (Billings  &  Durlach,  2008),  with  minor  changes  to  the 
reconnaissance  missions  to  allow  for  measurement  of  mission  completion  times. 

Before  beginning  training,  participants  completed  several  pre-task  measures.  Then 
participants  underwent  operator  training  using  the  OCU,  where  they  learned  how  to  control  the 
MAV  and  navigate  through  the  simulated  environment.  As  part  of  their  initial  training, 
participants  reviewed  a  manual  and  completed  six  practice  exercises.  After  successful 
completion  of  these  practice  exercises,  participants  completed  five  missions.  Three  missions 
were  flight  skill  missions,  where  participants  were  required  to  navigate  the  simulated  MAV 
following  landmarks  and  avoiding  obstacles.  Two  were  reconnaissance  missions,  where 
participants  had  to  locate  and  photograph  targets  using  the  simulated  MAV.  The  first  three 
missions  occurred  in  the  same  environment  where  initial  pratice  occurred,  while  the  last  two 
occurred  in  a  novel  environment.  After  each  training  mission,  participants  were  given  the  NASA 
Task  Load  Index  (NASA  TLX)  to  measure  subjective  workload.  Additional  perfonnance 
measures  included  mission  completion  times  and  number  of  collisions. 

Findings: 

Generally,  the  findings  replicated  the  pattern  of  results  from  the  previous  research.  The 
game  controller  configuration  that  was  used  in  Billings  and  Durlach  (2008)  led  to  significantly 
better  perfonnance  than  the  mouse  on  training  missions.  Findings  also  suggested  that  completion 
time  for  reconnaissance  missions  is  a  more  discriminating  performance  metric  than  number  of 
targets  detected  or  number  of  collisions  (which  were  used  in  our  prior  research).  Finally,  there 
were  several  findings  of  interest  related  to  game  controller  characteristics.  Results  indicated  that 
the  ability  to  control  movement  in  multiple  directions  simultaneously  yielded  significantly  better 
performance  than  single  direction  movement  for  flight  skill  missions,  but  user  control  over  speed 
failed  to  have  an  effect.  In  contrast,  during  reconnaissance  missions,  user  control  over  speed 
contributed  to  significantly  better  performance  than  no  control;  however,  control  in  only  a  single 
vs.  multiple  directions  failed  to  have  an  effect.  The  attentional  manipulation  failed  to  influence 
performance  in  either  type  of  mission.  Thus,  different  features  of  the  game  controller  contribute 
to  different  aspects  of  operator  performance.  Speed  control  was  a  significant  factor  for 
reconnaissance-related  tracking  tasks,  while  degrees  of  freedom  in  movement  direction  was  a 
significant  factor  for  navigation  tasks. 

Utilization  and  Dissemination  of  Findings: 

Temporal  measures  of  perfonnance  proved  sensitive  to  interface  manipulations  for  both 
flight  skill  and  reconnaissance  missions.  Different  aspects  of  input  device  control  affected 
performance  of  the  different  missions  in  different  ways.  The  results  demonstrate  the  importance 
of  matching  input  device  characteristics  to  task  characteristics.  Several  implications  for  the 
design  of  teleoperation  training  systems  and  the  design  of  input  controls  are  as  follows. 

Implications  for  Instructional  Design  for  Teleoperation 

•  To  achieve  effective  standards-based  training  for  teleoperation,  simulation-based  training 

should  include  a  series  of  progressively  more  challenging  missions.  Each  step  should 
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have  an  associated  performance  criterion  which  must  be  met  before  advancing  to  the  next 
more  challenging  level.  Missions  should  cover  the  entire  range  of  tasks  the  operator 
would  be  expected  to  perform.  Rationale:  In  prior  research  (Billings  &  Durlach,  2008),  as 
well  as  in  this  current  effort  following  this  guidelines,  students  learned  to  perform 
complex  simulated  missions  relatively  quickly. 

•  In  designing  standards-based  simulation  training  for  teleoperation,  temporal  measures  are 
very  discriminating  perfonnance  metrics  and  could  be  combined  with  required,  but  less 
sensitive  performance  metrics  to  assess  mastery.  Rationale:  During  the  course  of  our 
experiments  we  found  that  discrete  performance  measures  such  as  number  of  targets 
detected  or  number  of  collisions  were  relatively  insensitive  to  manipulation  of  the  input 
device.  They  may  represent  a  minimum  requirement  for  standards-based  training  (e.g.,  a 
mission  must  be  completed  without  any  collisions);  however,  to  discriminate  different 
degrees  of  mastery  above  and  beyond  these  essential  requirements,  mission  completion 
criterion  times  are  appropriate. 

•  The  possibility  that  experienced  video-game  players  may  perform  better  in  teleoperation 
should  be  further  investigated.  Rationale:  We  have  consistently  found  that  experienced 
video-game  players  perform  teleoperation  in  simulation  better  than  nonplayers;  however, 
it  is  unknown  whether  this  generalizes  to  live  environments. 


Implications  for  the  Design  of  Teloperation  Input  Controls 

•  Matching  user  input  device  characteristics  to  required  task  characteristics  improves 
teleoperation  performance.  The  entire  gamut  of  tasks  to  be  performed  must  be  considered 
to  maximize  the  benefits.  Rationale:  We  found  that  different  aspects  of  input  device 
control  affected  different  tasks  differently.  The  ability  to  maneuver  in  multiple  directions 
at  one  time  affected  route  following,  but  not  target  photography.  Conversely,  the  ability 
to  control  speed  affected  target  photography  but  not  route  following. 

•  The  specific  way  in  which  an  input  device  capability  is  implemented  and  not  just  the 
capability  itself  is  important,  because  the  implementation  may  affect  performance  on 
other  tasks.  Rationale:  We  found  (across  experiments)  that  the  ability  to  maneuver  in 
multiple  directions  at  a  time  facilitated  route  following.  Depending  on  this  ability  was 
implemented  in  the  user  interface,  however,  it  affected  performance  on  other  tasks  such 
as  photographing  targets. 
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Introduction 

Unmanned  aerial  systems  (UASs)  have  proven  their  ability  to  assist  in  a  wide  range  of 
military  missions,  especially  reconnaissance.  One  type  of  UAS  currently  under  development  is 
the  Class  I  micro -unmanned  aerial  vehicle  (MAV;  Coffrey  &  Montgomery,  2002;  Carey,  2007). 
MAVs  are  man-portable  and  have  the  potential  to  assist  small  military  units  in  local 
reconnaissance,  surveillance,  and  target  acquisition.  In  fact,  U.S.  Army  infantry  brigade  combat 
teams  are  scheduled  to  be  equipped  with  MAVs  by  201 1.  As  this  type  of  UAS  is  deployed  at  the 
small  unit  level,  new  training  requirements  will  need  to  be  developed,  for  training  of  operators, 
maintainers,  and  leaders.  MAV  operators  will  need  to  be  trained  on  system  operability  and 
maintenance,  while  commanders  will  need  training  on  system  utilization  (Durlach,  2007).  From 
an  operator  standpoint,  human-system  interaction  should  take  into  account  human  capabilities 
and  limitations  and  will  require  efficient  and  systematic  training  procedures.  Unfortunately,  the 
military  has  focused  more  on  developing  the  technology  than  on  designing  effective  human- 
robot  interaction  components  and  training  protocols  (Billings  &  Durlach,  2008).  Therefore, 
research  efforts  need  to  address  the  optimization  of  human-machine  interfaces  for  these  UAS 
operators,  as  well  as  the  proper  design  of  training  procedures,  performance  metrics,  and 
standards. 


Background 

Due  to  the  rapidly  approaching  training  demand,  we  previously  developed  simulation- 
based  operator  training  exercises  to  evaluate  the  usefulness  of  various  performance  measures  for 
their  ability  to  contribute  to  a  standards-based  simulation  training  curriculum  (Durlach, 
Neumann,  &  Billings,  2008;  Billings  &  Durlach,  2008).  During  the  course  of  this  prior  research, 
we  examined  the  impact  of  operator  interface  features  on  performance  and  found  that  for  flight 
skill  missions,  temporal  measures  (e.g.,  time  to  complete  a  mission)  were  most  sensitive  to  the 
manipulated  interface  features  (i.e.,  different  mouse  and  game  controller  input  device 
configurations).  In  contrast,  the  number  of  targets  detected  in  a  fixed-time  simulated 
reconnaissance  mission  failed  to  be  affected  by  the  manipulated  features.  One  purpose  of  the 
current  research  project  was  therefore  to  examine  whether  a  temporal  measure  for 
reconnaissance  missions  would  prove  more  sensitive  for  training  purposes.  Another  goal  was  to 
determine  why  certain  factors  of  our  previous  interface  manipulations  affected  perfonnance. 

Prior  Research  Findings 

Two  prior  investigations  were  conducted  with  the  simulation-based  MAV  training 
platform.  In  our  first  experiment  (Durlach,  Neumann,  &  Billings,  2008),  one  finding  was  that 
trainees  using  a  dual-thumb  stick  game  controller  (GC)  as  an  input  device  completed  skill 
missions  more  quickly  than  those  who  used  a  mouse  (M)  to  select  on-screen  icons  representing 
different  directions  of  MAV  movement.  There  were  several  ways  in  which  the  GC  configuration 
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differed  from  the  M  configuration,  and  any  one  of  these  could  have  contributed  to  the 
performance  difference,  one  of  which  is  spatial  learning. 

In  our  first  experiment  (Durlach,  Neumann,  &  Billings,  2008),  the  user  interface,  or 
simulated  operator  control  unit  (OCU),  was  equipped  with  streaming  video  of  the  virtual 
environment  in  which  the  simulated  MAV  flew  (as  captured  by  the  MAV’s  forward-looking  or 
downward-looking  fixed  cameras).  Visual  attention  to  this  sensor  window  was  necessary  to  see 
the  environment,  find  buildings,  avoid  obstacles,  detect  targets,  and  so  on.  Trainees  using  the  GC 
could  focus  all  their  visual  attention  on  this  window,  because  they  could  control  all  the  MAV 
functions  by  feel  with  the  GC.  In  contrast,  trainees  using  the  M  had  to  continuously  shift  their 
visual  attention  between  the  sensor  window  and  the  on-screen  icons  they  needed  to  select  to 
control  the  MAV  functions.  Because  participants  conducted  initial  training  (practice)  in  the  same 
environment  in  which  mission  performance  measures  were  taken,  it  is  possible  that  trainees  in 
the  GC  condition  may  have  learned  the  spatial  layout  of  the  simulated  village  better  during  this 
practice  period  than  the  trainees  in  the  M  condition.  Better  spatial  learning  may  have  allowed 
them  better  ability  to  anticipate  obstacles  and  use  landmarks  to  maneuver. 

In  our  second  experiment  (Billings  &  Durlach,  2008),  we  examined  the  possibility  that 
the  perfonnance  difference  between  GC  and  M  could  be  attributed  to  differences  in  spatial 
learning,  resulting  from  the  different  demands  the  two  input  devices  made  on  visual  attention. 
Two  missions  were  completed  in  a  virtual  environment  different  from  that  used  in  the  other 
missions.  In  this  way,  we  could  determine  if  the  performance  differences  between  the  GC  and  M 
transferred  to  a  new  environment,  or  if  performance  was  contingent  on  spatial  learning  of  the 
practice  environment.  Results  indicated  that  spatial  learning  was  not  a  likely  explanation, 
however  (Billings  &  Durlach,  2008).  The  GC  vs.  M  performance  difference  was  evident  both 
during  initial  training  and  during  the  transfer  test  in  a  novel  environment.  Because  neither  the  GC 
or  M  conditions  had  any  opportunity  to  learn  the  spatial  relations  of  the  novel  transfer 
environment,  differences  in  spatial  learning  during  training  could  not  underlie  the  performance 
differences. 

Differences  in  the  deployment  of  visual  attention  might  still  account  for  the  performance 
differences  observed  with  the  GC  and  M,  although  not  through  its  impact  on  spatial  learning.  It 
is  possible  that  the  requirement  to  continuously  shift  visual  attention  in  the  M  condition  affects 
performance  through  an  on-line  mechanism,  such  as  increased  demand  on  visual  working 
memory.  In  the  present  research,  we  attempted  to  assess  this  possibility,  by  isolating  the  visual 
attention  factor  across  input  devices.  In  our  initial  experiments,  the  GC  provided  the  ability  to 
maneuver  the  MAV  in  multiple  directions  simultaneously  (i.e.,  parallel  input  of  directional 
commands)  and  to  control  the  speed  of  the  MAV  anywhere  between  zero  and  six  knots.  These 
capabilities  were  enabled  via  the  thumb  sticks  such  that  the  direction  of  displacement  of  each 
stick  from  its  center  determined  two  axes  of  movement,  and  the  extent  of  the  displacement 
determined  speed.  In  contrast,  the  M  supported  the  ability  to  input  maneuver  commands  only  in 
one  axis  at  a  time  (e.g.,  up  or  forward,  but  not  both),  and  speed  was  controlled  by  the  system,  not 
the  operator.  The  M  condition  required  serial  input  of  all  movement  commands.  To  test  whether 
different  demands  on  visual  attention  accounted  for  the  performance  effects  observed,  we  re¬ 
engineered  GC  functioning  so  that  only  one  axis  of  movement  could  be  controlled  at  a  time  and 
the  system  controlled  the  speed  (see  GC2  in  Table  1).  In  this  way,  the  only  difference  in 
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functioning  between  the  GC2  condition  and  the  M  condition  (other  than  the  physical  device)  was 
that  attention  was  divided  between  the  sensory  imagery  and  the  navigation  controls  in  the  M 
condition,  and  attention  was  focused  on  the  sensory  imagery  in  the  GC2  condition. 


Table  1.  Conditions  and  the  Comparisons  Required  to  Test  the  Effects  of  Divided  Attention, 
Axis  Control,  and  Speed  Control 


Input  Device  Characteristics 

Condition 

Attention 

Axis  of  Control 

Speed  Control 

Physical  Input  Device 

M 

Divided 

One  at  a  time. 

System 

Mouse 

(same  as  Billings  &  Durlach,  2008) 

GC1 

Focused 

Multiple  axes 
simultaneously. 

User 

Game  Controller 

(same  as  Billings  &  Durlach,  2008). 

GC2 

Focused 

One  at  a  time. 

System 

Game  Controller 

GC3 

Focused 

Multiple  axes 
simultaneously. 

System 

Game  Controller 

GC4 

Focused 

One  at  a  time. 

User 

Game  Controller 

Group 

Comparison 

Factors  that  were  isolated  by  the  comparison 

M  vs.  GC1 

Comparison  made  in  previous  experiments,  factors  confounded 

M  vs.  GC2 

Isolates  effect  of  visual  attention 

GC1  +GC4 

vs. 

GC3  +  GC2 

Isolates  effect  of  speed  control,  collapsing  over  axis  control 

GC1  +GC3 

vs. 

GC4  +  GC2 

Isolates  effect  of  axis  control,  collapsing  over  speed  control 

GC  functioning  was  also  manipulated  across  three  additional  conditions  to  allow 
determination  of  the  separate  effects  of  axis  control  and  speed  control.  To  change  the  GC  to  one- 
direction  at  a  time  mode,  the  dominant  direction  of  displacement  on  a  thumb  stick  was  translated 
into  movement  in  the  dominant  direction  only.  To  change  the  GC  to  system  speed  control,  the 
extent  of  thumb  stick  displacement  was  made  irrelevant.  The  MAV  started  and  stopped  with 
some  inertia.  Once  given  the  command  to  move,  it  gradually  sped  up  to  six  knots  and  then  stayed 
at  that  speed  until  a  new  command  was  issued,  regardless  of  how  far  the  thumb  stick  was 
displaced  from  center.  The  various  resulting  conditions  of  the  present  experiment  are  listed  in 
Table  1,  along  with  the  comparisons  required  to  detennine  the  effects  of  visual  attention,  axis 
movement  control,  and  speed  control  on  performance.  For  GC2,  the  MAV  could  only  move  in 
one  direction  at  a  time,  and  speed  was  controlled  by  the  system.  For  GC3,  movement  in  multiple 
axes  was  enabled,  and  speed  was  controlled  by  the  system.  For  GC4,  movement  was  in  only  one 
direction  at  a  time,  and  the  flight  speed  was  controlled  by  the  user. 
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Method 


Participants 

Twenty-five  male  and  25  female  participants  from  the  University  of  Central  Florida 
completed  this  experiment  in  exchange  for  college  course  credit  or  monetary  compensation.  The 
mean  age  was  19.5  years  (range  18  to  35).  Four  other  participants  (all  female)  failed  to  pass  the 
initial  performance  criteria  and  were  excused  from  the  experiment.  All  participants  signed  an 
informed  consent  before  beginning  the  experiment.  Each  experimental  condition  consisted  of  10 
participants  and  had  an  equal  number  of  males  and  females  to  account  for  possible  gender 
differences  related  to  video  game  experience  and  spatial  ability. 

Materials 

A  combination  of  subjective  measures  and  questionnaires  were  used,  including  a 
demographics  survey  (see  Appendix  B),  a  training  evaluation  worksheet  (See  Appendix  C),  the 
Hidden  Patterns  Test  (ETS,  1976),  and  an  electronic  form  of  the  NASA  Task  Load  Index  (TLX; 
Hart  &  Staveland,  1988).  The  demographics  survey  was  a  paper-based  questionnaire  requesting 
information  about  gender,  remote  control  experience,  video  game  skills,  and  computer 
experience.  The  training  evaluation  worksheet  presented  participants  with  a  screenshot  of  the 
OCU  and  required  participants  to  match  the  components  in  the  interface  with  their  functions. 

This  worksheet  verified  comprehension  of  the  MAV  interface  and  operating  procedures.  The 
Hidden  Patterns  Test  was  a  paper-based  measure  of  spatial  ability  and  required  participants  to 
find  simple  patterns  within  more  complex,  distracting  patterns  (ETS,  1976).  This  particular 
measure  of  spatial  ability  was  used  because  Durlach,  Neumann,  and  Billings  (2008)  found  that 
results  on  this  test  correlated  with  several  dependent  measures  of  perfonnance  that  were  used  in 
the  current  research.  The  NASA  TLX  was  administered  on  a  desktop  computer,  where  workload 
scores  were  automatically  calculated.  Participants  were  required  to  rate  their  perceived  workload 
on  six  different  dimensions,  including  mental  demand,  physical  demand,  temporal  demand, 
performance,  effort,  and  frustration. 

The  experimenter  followed  a  script  carefully  to  ensure  that  all  experimental  procedures 
were  administered  correctly  and  that  all  participants  received  the  same  instructions  (See 
Appendix  D).  A  training  manual  provided  detailed  information  about  the  simulation  and 
instructed  participants  how  to  operate  the  simulated  MAV.  Participants  read  through  the  training 
manual,  which  led  them  through  training  exercises.  A  full  copy  of  the  participant  training  manual 
can  be  found  in  Billings  and  Durlach  (2008). 

Apparatus 

The  simulation  required  the  use  of  three  computers.  The  OCU,  the  synthetic  terrain 
database,  and  the  MAV  simulator  were  run  on  one  computer.  A  computer  ran  One-SAF  Testbed 
(OTB)  version  2.5,  which  allowed  insertion  of  entities  (people  and  vehicles)  into  the  synthetic 
environment  via  a  network.  Finally,  a  third  computer  was  used  to  administer  the  NASA  TLX. 
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The  configuration  of  the  OCU  screen  is  illustrated  in  Figure  1,  and  was  identical  to  that 
used  by  Billings  &  Durlach  (2008).  It  had  a  window  with  streaming  camera  imagery,  an 
altimeter,  a  mission  timer,  task  bar  icons,  a  heading  tape,  an  overhead  map  view,  and  a  control 
pad  that  provided  visual  feedback  to  the  operator.  The  simulated  MAV  was  based  on 
Honeywell’s  Class  I  t-MAV  prototype,  which  incorporated  a  ducted  fan  design  that  allowed  for 
vertical  take-off  and  landing,  hovering  in  mid-air,  and  rotating  in  place  (Carey,  2007).  The  flight 
model  included  some  basic  inertial  properties,  and  like  the  t-MAV  prototype  limited  speed  to  six 
knots  in  manual  mode,  and  tilted  forward  in  forward  flight  1°  for  every  knot  of  forward  speed. 
The  simulated  MAV  was  equipped  with  two  fixed  cameras,  one  pointed  down,  and  the  other 
pointed  forward.  Only  one  camera  view  appeared  on-screen  at  a  time,  and  participants  switched 
views  using  their  respective  input  device  (pressing  a  button  on  the  GC,  and  clicking  an  on-screen 
icon  with  the  M). 


Heading  tape 


Mission  timer 


Altimeter 


Video  sensor  imagery 


Control  pad 


Overhead  map 


Task  bar  icons 


Figure  1.  Layout  of  the  Operator  Control  Unit  (OCU). 
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Participants  sat  at  a  desk  and  piloted  the  simulated  MAV  using  either  a  Logitech  dual 
thumb-stick  GC  (See  Figure  2)  or  a  standard  two-button  M  with  a  scroll  wheel.  Participants 
using  the  M  clicked  on  the  desired  icon  on  the  OCU  to  give  commands  to  the  MAV.  Participants 
using  the  GC  manipulated  the  controller  to  issue  commands  to  the  MAV.  There  were  four 
different  GC  conditions,  as  outlined  in  Table  1.  Input  device  set-up  was  manipulated  as  a 
between-group  variable,  i.e.,  each  participant  had  the  same  configuration  throughout  the 
experiment.  Regardless  of  the  input  configuration  used,  the  same  visual  feedback  appeared  on 
the  control  pad  (e.g.,  when  directed  to  move  forward,  the  OCU  forward-arrow  icon  lit  up). 


Press  to  take  off  and  land 


Activate 

camera 

selection 

icons 

(release 

Ito  select) 


Figure  2.  Logitech  GC  mapping  of  commands  for  the  GC  conditions. 


Procedure 

Participants  were  assigned  to  one  of  the  five  input  device  configurations  outlined  in  Table 
1 .  They  each  signed  an  informed  consent  form  and  then  completed  a  demographics  survey  and 
the  Hidden  Patterns  Test.  Participants  were  then  trained  to  operate  the  simulated  MAV,  with  the 
aid  of  a  manual  which  guided  them  through  six  hands-on  practice  exercises.  For  the  first  four 
practice  exercises,  participants  followed  simple  consecutive  movement  directions  such  as  take¬ 
off,  fly  forward,  rotate,  and  land.  These  exercises  allowed  the  participant  to  practice 
manipulating  the  controls  of  the  input  device.  Practice  Exercises  5  and  6  were  flight  skill 
exercises  and  required  participants  to  pilot  the  MAV  along  predetermined  paths,  avoiding  poles 
and  other  obstacles.  Each  of  the  practice  exercises  required  completion  within  a  specific 
criterion  time  and  without  any  collisions  before  moving  on  to  the  next  practice  exercise.  If  they 
collided  with  anything,  participants  were  restarted  from  the  beginning  of  the  exercise. 

Participants  were  given  five  attempts  to  successfully  complete  each  of  the  practice  exercises 
without  collisions.  Four  female  participants  who  were  unable  to  do  this  were  dismissed  from  the 
experiment. 

The  practice  exercises  were  completed  in  a  training  environment  that  simulated  a  military 
operations  in  urban  terrain  (MOUT)  training  area  at  Fort  Polk,  LA,  which  included  trees, 
buildings,  and  other  potential  obstacles.  For  Practice  Exercises  1,  2,  3,  and  4,  participants  were 
required  to  perfonn  various  actions  with  the  MAV,  such  as  taking  off,  landing,  capturing  a 
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photograph,  and  following  a  series  of  oral  commands.  These  exercises  ensured  that  participants 
had  learned  the  mappings  of  the  input  device.  For  Practice  Exercise  5,  participants  manually 
piloted  the  MAV  around  a  gray  pathway  while  remaining  to  the  left  of  four  red  poles,  and  then 
they  landed  in  a  designated  landing  zone.  This  exercise  had  to  be  completed  within  three 
minutes  and  50  seconds.  For  Practice  Exercise  6,  participants  manually  piloted  the  MAV 
through  an  obstacle  course  similar  to  a  slalom  course,  where  participants  had  to  weave  between 
red  and  green  poles.  At  the  end  of  the  course,  they  were  also  required  to  photograph  a  target 
before  landing  in  the  designated  landing  zone.  Participants  had  five  minutes  to  complete  this 
exercise.  The  time  criteria  were  set  in  order  to  make  sure  that  participants  were  able  to 
demonstrate  competency  in  the  tasks  in  a  timely  fashion.  Full  details  of  the  practice  exercises 
can  be  found  in  the  Experimenter’s  Script  (Appendix  D)  as  well  as  in  Billings  and  Durlach 
(2008).  Performance  data  was  recorded  for  each  of  these  practice  exercises. 

After  successful  completion  of  the  practice  exercises,  the  training  evaluation  worksheet 
and  NASA  TLX  were  administered  to  participants.  Next,  participants  completed  five  missions 
during  which  their  performance  data  were  recorded.  Three  missions  were  flight  skill  tasks.  For 
these,  participants  had  to  navigate  around  obstacles  while  following  a  predetermined  path.  The 
other  two  missions  were  reconnaissance  tasks.  For  these,  participants  had  to  photograph  a  given 
number  of  targets.  The  first  three  missions  (two  skill  missions,  one  reconnaissance  mission)  were 
conducted  in  the  Fort  Polk,  LA,  MOUT  area,  whereas  the  last  two  missions  (one  skill  mission, 
one  reconnaissance  mission)  were  conducted  in  a  new  simulated  environment,  a  MOUT  site 
from  Fort  Benning,  GA.  For  all  missions,  the  primary  dependent  variable  was  completion  time. 
Participants  who  committed  a  collision  during  the  course  of  a  mission  were  instructed  to  restart, 
and  all  mission  times  used  were  for  collision-less  missions. 

Missions  1  and  2  required  participants  to  navigate  courses  designated  by  poles  throughout 
the  training  environment  of  Fort  Polk,  LA.  In  Mission  1,  participants  followed  a  road  around  the 
outskirts  of  the  MOUT  site  (resembling  a  racetrack).  In  Mission  2,  participants  navigated  a 
slalom  course  in  which  they  weaved  through  poles.  These  skill  missions  were  similar  to  the  last 
two  practice  exercises  (Practice  Exercise  5  and  Practice  Exercise  6).  The  only  difference  was  that 
during  the  practice  exercises,  participants  were  able  to  track  the  location  of  the  MAV  along  a 
visible  route  on  the  satellite  map.  During  the  missions,  this  path  was  not  visible. 

For  Mission  3,  participants  were  given  pictures  of  various  targets  and  were  instructed  to 
fly  around  the  environment  to  find  and  photograph  six  (stationary)  targets,  and  then  to  land  the 
MAV  back  at  the  launch  point.  Participants  were  allowed  a  maximum  of  15  minutes.  Mission  4 
was  a  skill  mission  that  involved  navigation  around  poles  and  obstacle  avoidance  in  a  new 
location,  which  was  the  Fort  Benning,  GA-MOUT  environment.  Mission  5  was  a  reconnaissance 
mission  also  performed  in  the  new  location  of  the  Fort  Benning,  GA-MOUT  environment. 

Similar  to  Mission  3,  participants  were  required  to  locate  and  photograph  six  targets  with  a 
maximum  allowed  time  of  15  minutes;  the  targets  were  new  and  five  were  stationary  whereas  the 
sixth  was  moving. 

Participants  completed  the  NASA  TLX  following  each  of  the  missions.  After  completing 
Mission  5,  participants  were  debriefed  and  compensated  for  their  time.  Computerized  logs  were 
kept  of  each  participant’s  mission  times  and  workload  scores. 
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Results 


Data  Analysis  Plan 

Because  our  previous  research  showed  some  relation  between  video  game  experience 
(VGE)  and  performance,  as  well  as  between  spatial  ability  and  performance,  these  measures 
were  used  as  covariates.  A  single  value  was  used  for  (VGE)  by  standardizing  and  summing 
three  highly  correlated  demographic  questions  relating  to  video  game  usage  and  skill  into  a 
single  score.  The  score  from  the  Hidden  Patterns  Test  was  used  for  spatial  ability.  The  dependent 
variables  of  interest  were  mission  completion  times  and  NASA  TLX  scores.  When  covariates  are 
not  mentioned,  they  failed  to  have  a  significant  effect. 

For  each  dependent  measure,  three  separate  analyses  were  conducted.  First,  GC1  and  M 
were  compared  to  check  whether  the  performance  differences  found  in  our  previous  research  had 
been  replicated,  as  these  input  device  configurations  were  the  same  as  those  used  in  our  prior 
research.  Next,  GC2  and  M  were  compared.  These  conditions  differed  only  in  the  aspect  of 
visual  attention.  Better  perfonnance  in  GC2  than  M  would  indicate  that  having  to  divide 
attention  between  the  sensor  window  and  the  control  pad  is  detrimental  to  performance.  Finally, 
a  2  x  2  analysis  of  covariance  (ANCOVA)  was  conducted  using  the  data  from  GC 1  -  GC4,  with 
factors:  movement  control  (one  vs.  multiple)  and  speed  control  (system  vs.  user). 

For  the  temporal  data  for  each  of  the  three  comparisons  described  above,  we  conducted 
two  separate  analyses,  one  for  flight  skill  missions,  and  one  for  reconnaissance  missions.  Thus, 
for  the  flight  skill  mission  analysis  there  were  five  repeated  measures  (Practice  5,  Practice  6, 
Mission  1,  Mission  2,  and  Mission  4).  The  practice  exercises  were  included  in  the  analyses 
because  they  were  identical  and  employed  the  same  skills  and  characteristics  of  Missions  1,  2, 
and  4.  In  this  way,  performance  across  these  similar  missions  could  be  analyzed.  For  the 
reconnaissance  mission  analysis  there  were  two  repeated  measures  (Missions  3  and  5). 

Mission  Completion  Times — Flight  Skill  Missions 

The  temporal  perfonnance  data  for  the  flight  skills  missions  were  examined  first.  Table  2 
shows  the  mean  completion  times  and  standard  deviations  for  Practice  5,  Practice  6,  Mission  1, 
Mission  2,  and  Mission  4  for  the  different  input  device  configurations.  In  addition,  mean 
completion  times  for  collapsed  conditions  are  shown.  GC1  and  GC4  isolated  user  control  of 
speed.  GC3  and  GC2  isolated  system  control  of  speed.  By  collapsing  these  groups,  comparisons 
between  user-controlled  speed  and  system-controlled  speed  can  be  made.  In  addition,  GC1  and 
GC3  isolated  participants  who  could  navigate  in  multiple  directions  simultaneously.  GC4  and 
GC2  isolated  participants  who  could  control  movement  in  only  one  direction  at  a  time.  In  this 
way,  multiple  versus  single  directional  control  could  be  analyzed. 
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Table  2.  Mean  Completion  Times  in  Seconds  (Standard  Deviations)  for  Flight  Skill  Missions 


Flight  Skill  Missions 


Input  Device 
Configuration 

Practice  5 

Practice  6 

Mission  1 

Mission  2 

Mission  4 

M 

GC1 

GC2 

GC3 

GC4 

212.57  (9.13) 
190.79  (15.11) 
212.71  (7.54) 
208.64  (16.39) 
210.17(10.72) 

263.14  (19.48) 
218.89  (26.79) 
261.79  (27.87) 
245.88  (34.71) 
259.42  (29.49) 

211.20  (19.40) 
187.51  (16.61) 
209.07  (22.72) 
189.58  (18.11) 
202.08  (15.83) 

254.76  (32.51) 
227.60  (47.68) 
246.96  (33.41) 
240.79  (32.66) 
238.92  (34.88) 

282.79  (61.17) 
220.58  (29.08) 
259.29  (37.39) 
233.60  (50.79) 
256.00  (34.27) 

Speed  control  isolation 

GC1  +  GC4 
GC3  +  GC2 

200.48  (16.17) 
210.67(12.59) 

239.16(34.41) 
253.84  (31.71) 

194.80(17.47) 
199.33  (22.35) 

233.26  (41.07) 
243.87  (32.31) 

238.29  (35.87) 
246.44  (45.37) 

Directional  control  isolation 

GC1  +  GC3 
GC4  +  GC2 

199.72(17.87) 
211.44  (9.11) 

232.39  (33.20) 
260.61  (27.96) 

188.55  (16.94) 
205.58  (19.39) 

234.19  (40.35) 
242.94  (33.50) 

227.09  (40.83) 
257.64  (34.95) 

Mouse  vs.  GC1.  Figure  3  illustrates  the  mission  completion  times  across  flight  skill 
missions  for  participants  in  the  M  and  GC 1  configurations.  These  configurations  were  identical 
to  the  configurations  used  in  our  prior  research  (Billings  &  Durlach,  2008).  A  2  x  5  mixed  design 
ANCOVA  was  used  to  analyze  data  from  M  and  GC1,  with  the  repeated  measure  representing 
skill  mission.  Mauchly’s  test  indicated  that  the  assumption  of  sphericity  had  been  violated  (chi- 
square  =  32.20 ,p<  .005),  and  therefore  degrees  of  freedom  were  corrected  using  Greenhouse- 
Geisser  estimates  of  sphericity  (epsilon  =  .54).  There  were  significant  differences  in  average 
times  to  complete  the  different  missions,  F(2.17,  34.73)  =  20.84,/?  <  .005,  qp  =  .57,  with  both 
groups  showing  shorter  completion  times  for  Practice  5  and  Mission  1  than  for  the  others.  Of 
more  interest,  the  main  effect  of  condition  was  significant,  F(  1,  16)  =  14.60,/?  <  .005,  qp  =  .48, 
replicating  our  prior  results  that  mission  times  were  shorter  in  GC  1  than  M. 
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Mean  Completion  Times  (in  Seconds)  Across  Flight  Skill  Missions 
Vertical  bars  denote  0.95  confidence  intervals 


Figure  3.  Mean  completion  times  (in  seconds)  across  flight  skill  missions,  for  GC1  and  M 
configurations. 


Attention.  A  2  x  5  mixed  design  ANCOVA  was  used  to  analyze  the  data  from  M  and 
GC2.  These  configurations  were  identical  except  that  M  required  participants  to  divide  visual 
attention,  whereas  GC2  allowed  participants  to  focus  their  visual  attention.  In  this  way,  any 
existing  performance  differences  between  these  two  configurations  could  be  attributed  to  an 
attentional  factor.  Figure  4  illustrates  the  mission  completion  times  across  flight  skill  missions 
for  participants  in  the  M  and  GC2  configurations.  Mauchly’s  test  indicated  that  the  assumption 
of  sphericity  had  been  violated  (chi-square  =  26.84 ,p<  .005),  so  degrees  of  freedom  were 
corrected  using  Greenhouse-Geisser  estimates  of  sphericity  (epsilon  =  .54).  Mission  was 
significant,  F(2.15,  34.40)  =  26.22 ,p<  .005,  qp  =  .62,  with  both  groups  showing  shorter 
completion  times  for  the  “racetrack”  missions  (Practice  5  and  Mission  1).  There  failed  to  be  a 
significant  effect  of  attention,  F(  1,  16)  =  0.60,/?  =  .45,  qp  =  .04.  These  results  suggest  that  the 
difference  in  flight  skill  completion  time  between  GC 1  and  M  (as  was  found  in  our  prior 
research)  was  not  due  to  different  demands  on  visual  attention. 
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Mean  Completion  Times  (in  Seconds)  Across  Flight  Skills  Missions 
Vertical  bars  denote  0.95  confidence  intervals 


Practice  5  Practice  6  Mission  1  Mission  2  Mission  4 

FLIGHT  SKILL  EXERCISE 


Figure  4.  Mean  completion  times  (in  seconds)  across  flight  skill  missions,  for  GC2  and  M 
configurations. 


Directions  and  speed  control.  A  2  x  2  x  5  (movement  ability  x  speed  control  x  skill 
mission)  mixed  design  ANCOVA  was  used  to  assess  the  effects  of  movement  control  and  speed 
control.  Mauchly’s  test  indicated  that  the  assumption  of  sphericity  had  been  violated  (chi-square 
=  59.00 ,p<  .005),  and  degrees  of  freedom  were  corrected  using  Greenhouse-Geisser  estimates 
of  sphericity  (epsilon  =  .64).  Mission  was  significant,  F( 2.56,  87.01)  =  47.00,  p  <  .005,  qp  =  .58, 
with  all  groups  showing  shorter  completion  times  for  the  “racetrack”  missions.  The  effect  of 
movement  control  was  also  significant,  F(  \ ,  34)  =  1 .01,  p  =  .01,  qp  =  .17  (See  Figure  5). 
Conditions  GC1  and  GC3,  which  could  move  in  multiple  directions  simultaneously,  had  shorter 
mission  times  overall  (M=  217.0  s)  than  conditions  GC2  and  GC4  (M=  235.03  s),  which  could 
move  in  only  one  direction  at  a  time.  Speed  control  failed  to  have  a  significant  impact,  F(  1,  34)  = 
1.18 ,p  =  .29,  r|p  =  .03  (See  Figure  6).  Mean  completion  time  was  222.2  s  for  GC1  and  GC4 
combined,  and  was  229.8  s  for  GC2  and  GC3  combined.  The  interaction  between  movement 
ability  and  speed  control  failed  to  be  significant,  F(  \ ,  34)  =  1.0 6,p  =  .31,  qp  =  .03). 
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Completion  Times  (in  Seconds)  Across  Flight  Skill  Mission  for 
Single  Versus  Multiple  Directional  Control 
Vertical  bars  denote  0.95  confidence  intervals 


Figure  5.  Mean  completion  time  (in  seconds)  across  flight  skill  missions  for  GC  configurations 
with  single  versus  multiple  directional  control. 
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Completion  Times  (in  Seconds)  Across  Flight  Skill  Mission  for 
User  Versus  System-Controlled  Speed 
Vertical  bars  denote  0.95  confidence  intervals 


Figure  6.  Mean  completion  time  (in  seconds)  across  flight  skill  missions  for  GC  configurations 
with  user  versus  system-controlled  speed. 

Mission  Completion  Times — Reconnaissance  Missions 

The  temporal  performance  data  for  the  reconnaissance  missions  were  examined  next. 
Table  3  shows  the  mean  completion  times  and  standard  deviations  for  Mission  3  and  Mission  5 
for  the  different  input  device  configurations,  as  well  as  the  collapsed  conditions.  Again,  GC1 
and  GC4  isolated  user  control  of  speed,  while  GC3  and  GC2  isolated  system  control  of  speed. 
Furthermore,  GC  1  and  GC3  isolated  movement  in  multiple  directions  simultaneously,  while 
GC4  and  GC2  isolated  movement  in  one  direction  at  a  time. 
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Table  3.  Mean  Completion  Times  in  Seconds  (Standard  Deviations)  for  Reconnaissance 
Missions 


Input  Device 
Configuration 

Reconnaissance  Missions 

Mission  3 

Mission  5 

M 

700.15  (153.35) 

740.59(128.77) 

GC1 

483.48  (175.49) 

432.04(192.19) 

GC2 

683.26  (183.67) 

608.12(219.74) 

GC3 

632.87  (195.63) 

529.26  (243.22) 

GC4 

494.84(150.06) 

491.26(198.15) 

Speed  control  isolation 

GC1  +  GC4 

489.16  (159.02) 

461.65  (192.40) 

GC3  +  GC2 

658.06  (186.48) 

568.69(229.19) 

Directional  control  isolation 

GC1  +  GC3 

558.17  (196.44) 

480.65  (219.10) 

GC4  +  GC2 

589.05  (189.71) 

549.69  (212.28) 

Mouse  vs.  GC1.  Figure  7  illustrates  mission  completion  times  for  the  reconnaissance 
missions  graphically.  A  2  x  2  (input  device  x  mission)  mixed  design  ANCOVA  indicated  lack  of 
difference  in  completion  times  for  Missions  3  and  5,  F(l,  16)  =  .08 ,p  =  .78,  rjp  =.01.  More 
importantly,  there  was  a  significant  effect  due  to  input  device,  F(  1,  16)  =  16.15,/?  <  .005,  r|p  = 

.50.  As  can  be  seen  in  Figure  7,  GC1  completed  reconnaissance  missions  more  quickly  than  M. 

In  our  prior  experiments,  there  were  no  significant  perfonnance  effects  detected  for  our 
reconnaissance  missions  (Billings  &  Durlach,  2008);  however,  we  had  used  a  different 
performance  measure,  number  of  targets  detected.  The  present  results  therefore  suggest  that  the 
temporal  measure  used  here  is  more  sensitive  to  performance  differences. 
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Mean  Completion  Times  (in  Seconds)  Across  Reconnaissance  Missions 
Vertical  bars  denote  0.95  confidence  intervals 
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Figure  7.  Mean  completion  times  (in  seconds)  across  reconnaissance  missions,  for  GC1  and  M 
configurations. 


Mission  3  Mission  5 

RECONNAISSANCE  EXERCISE 


Attention.  A  2  x  2  (attention  x  mission)  mixed  design  ANCOVA  examined  data  from  M 
and  GC2.  Mission  failed  to  have  a  significant  impact  F(  1,  16)  =  0.35,  p  =  .56,  r)p  =  .02,  as  did 
attention,  F{  1,  16)  =  1.08,/?  =  .31,  r|p  =  .06.  These  results  suggest  that  the  difference  in 
reconnaissance  mission  completion  time  between  GC1  and  M  (as  found  in  the  current  research, 
as  well  as  our  prior  research)  was  not  due  to  different  demands  on  visual  attention. 
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Mean  Completion  Times  (in  Seconds)  Across  Reconnaissance  Missions 
Vertical  bars  denote  0.95  confidence  intervals 
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Figure  8.  Mean  completion  times  (in  seconds)  across  reconnaissance  missions,  for  GC2  and  M 
configurations. 
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RECONNAISSANCE  EXERCISE 


Direction  and  speed  control.  A  2  x  2  x  2  (movement  control  x  speed  control  x 
reconnaissance  mission)  mixed  design  ANCOVA  detected  a  significant  effect  of  mission,  F(  1, 
34)  =  5.82 ,p  =  .02,  r|p  =  .15.  When  averaged  across  the  four  conditions  (GC1  -  GC4),  Mission  3 
was  completed  more  quickly  than  Mission  5.  The  effect  of  movement  control  failed  to  be 
significant,  F(  1,  34)  =  0.52,  p  =  .48,  qp  =  .02;  See  Figure  9  for  graphical  illustration.  However, 
the  effect  of  speed  control  was  significant,  F{  1,  34)  =  4.45,  p  =  .04,  qp  =  .12.  (See  Figure  10). 
User  control  of  speed  in  GC1  and  GC4  allowed  faster  reconnaissance  mission  completion  time 
(M=  486.82s)  than  did  system  control  for  GC2  and  GC3  (M=60 1.96s).  There  was  no  evidence 
for  an  interaction  between  movement  control  and  speed  control,  F(  1,  34)  =  .01  ,p  =  .92,  qp  =  .00). 
VGE  was  found  to  be  a  significant  covariate,  F(  1,  34)  =  6.56,  p  =  .02,  qp  =  .16. 
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Mean  Completion  Times  (in  Seconds)  Across  Reconnaissance  Missions 
for  Single  Versus  Multiple  Directional  Control 
Vertical  bars  denote  0.95  confidence  intervals 
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Figure  9.  Mean  completion  time  (in  seconds)  across  reconnaissance  missions  for  GC 
configurations  with  single  versus  multiple  directional  control. 
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Mean  Completion  Times  (in  Seconds)  Across  Reconnaissance  Missions 
for  User  Versus  System-Controlled  Speed 
Vertical  bars  denote  0.95  confidence  intervals 


Figure  10.  Mean  completion  time  (in  seconds)  across  reconnaissance  missions  for  GC 
configurations  with  user  versus  system-controlled  speed. 

NASA  TLX  Workload  Analyses 

Data  analysis  of  workload  measures  followed  the  same  strategy  as  above,  except  each 
mission  was  analyzed  separately  because  of  nonparallelism  in  the  covariates  across  the  missions. 
Table  4  gives  the  means  for  each  condition  and  comparison  for  the  flight  skill  missions.  Table  5 
gives  the  analogous  data  for  reconnaissance  missions. 

Flight  skill  missions.  Table  4  lists  the  means  for  the  flight  skill  missions.  In  general, 
workload  was  rated  lower  in  GC  1  than  M,  and  the  difference  was  significant  for  ratings  given 
after  Practice,  F(l,  16)  =  5.97,/?  =  .03,  r|p  =  .27,  Mission  2,  F( T,  16)  =  5.06,/?  =  .04,  r|p  =  .24, 
and  Mission  4,  F(l,  16)  =  9.36,/?  =  .01,  r|p  =  .37.  Higher  VGE  was  associated  with  lower 
workload  scores,  and  this  covariation  was  significant  for  Missions  2  and  4.  Pearson  r’s  =  -.57, 

F{  1,  16)=  12.77,/?  <  .005  and -.44,  F(l,  16)  =  6.10,/?  =  .03,  for  Missions  2  and  4,  respectively. 
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Table  4.  Mean  Rated  Workload  (Standard  Deviation)  for  Flight  Skill  Missions 


Flight  Skill  Missions 

Input  Device 
Configuration 

Practice 

Mission  1 

Mission  2 

Mission  4 

M 

61.3  (18.8) 

41.5  (16.7) 

49.9  (20.8) 

62.2(18.2) 

GC1 

40.9  (25.7) 

27.6(16.6) 

31.7(17.4) 

34.6  (20.6) 

GC2 

59.9(12.9) 

47.4(16.7) 

50.6(19.5) 

48.3  (19.7) 

GC3 

49.2(17.0) 

38.3  (17.9) 

39.2(17.5) 

46.2  (23.7) 

GC4 

46.5  (19.6) 

36.4(17.1) 

43.3  (16.2) 

50.3  (20.9) 

GC1  +  GC4 

43.7  (22.4) 

32.0(17.0) 

37.5  (17.4) 

42.5  (21.7) 

GC3  +  GC2 

54.6(15.7) 

42.9(17.5) 

44.9(18.9) 

47.3  (21.2) 

GC1  +  GC3 

45.1  (21.6) 

32.9(17.6) 

35.5  (17.4) 

40.4  (22.4) 

GC4  +  GC2 

53.2(17.5) 

41.9(17.4) 

47.0(17.8) 

49.3  (19.8) 

With  respect  to  the  comparison  of  GC2  and  M,  there  failed  to  be  any  significant 
differences,  indicating  that  the  different  attentional  demands  of  these  two  conditions  had  no 
detectable  effects.  To  assess  whether  movement  control  or  speed  control  affected  workload,  the  2 
x  2  analysis  was  conducted  for  each  mission.  Movement  control  failed  to  have  a  significant 
effect  for  any  mission.  Likewise,  movement  control  failed  to  have  a  significant  impact  during 
Practice,  Mission  1  or  Mission  4.  For  Mission  2,  however,  participants  with  greater  movement 
control  reported  significantly  lower  workload,  ifiT,  34)  =  4.45,  p  =  .04,  r|p  =  .12.  For  this  2x2 
analysis,  higher  VGE  tended  to  be  associated  with  lower  workload  ratings,  and  was  a  significant 
covariate  for  each  mission  except  Mission  1.  For  Practice,  Pearson  r  =  -.38,  F(l,  34)  =  5.08,/?  = 
.03,  r|p  =  .13.  For  Mission  2,  Pearson  r  =  -.33,  F(l,  34)  =  4.54,/?  =  .04,  r|p  =  .12.  For  Mission  4, 
Pearson  r  =  -.41,  F(l,  34)  =  1 1.63,/?  <  .005,  r|p  =  .26.  Higher  Spatial  ability  also  tended  to  be 
associated  with  lower  workload  ratings;  but  this  relation  was  significant  only  for  Mission  4, 
Pearson  r=  .21,  F(l,  34)  =  5.17,/?  =  .03,  rf  =  .13. 

Reconnaissance  missions.  Table  5  shows  the  mean  workload  scores  for  the 
reconnaissance  missions.  Workload  was  rated  as  lower  in  GC1  than  M,  for  both  Mission  3,  F(l, 
16)  =  5.06,/?  =  .04,  r|p  =  .24,  and  Mission  5,  F(  1,  16)  =  5.74,/?  =  .03,  r|p  =  .26.  Higher  VGE  was 
associated  with  lower  workload  scores  and  was  significant  for  Mission  3,  Pearson  r  =  -.41,  F(l, 
16)  =  4.60,/?=  .05. 
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Table  5.  Mean  Rated  Workload  (Standard  Deviation)  for  Reconnaissance  Missions 


Input  Device 
Configuration 

Reconnaissance  Missions 

Mission  3 

Mission  5 

M 

57.4(18.1) 

63.3  (20.7) 

GC1 

38.  9  (20.2) 

38.3  (23.0) 

GC2 

64.6(17.3) 

57.9(16.3) 

GC3 

51.7  (22.7) 

50.1  (24.8) 

GC4 

52.9(18.9) 

51.9(23.0) 

GC1  +  GC4 

45.9  (20.4) 

45.3  (23.4) 

GC3  +  GC2 

58.2  (20.8) 

54.0  (20.8) 

GC1  +  GC3 

45.3  (21.9) 

44.4  (24.0) 

GC4  +  GC2 

58.8(18.7) 

54.9(19.7) 

With  respect  to  the  comparison  of  GC2  and  M,  there  failed  to  be  any  significant 
differences,  indicating  that  the  different  attentional  demands  of  these  two  conditions  had  no 
detectable  effects  on  workload  during  reconnaissance  missions.  To  assess  whether  movement 
control  or  speed  control  affected  workload,  the  2x2  analysis  was  conducted  for  each 
reconnaissance  mission.  The  only  significant  effect  detected  was  of  movement  control  in 
Mission  3,  F{  1,  34)  =  5.09, =  .03,  r|p  =  .13,  with  lower  workload  reported  by  participants  with 
the  ability  to  move  in  multiple  directions  at  one  time  compared  with  participants  who  could  only 
move  in  a  single  direction  at  a  time.  For  this  2x2  analysis,  higher  VGE  was  a  significantly 
associated  with  lower  workload  for  both  Mission  3,  Pearson  r  =  -.46,  F(l,  34)  =  9.70,  p  <  .005 
and  Mission  5,  Pearson  r  =  -.40,  F(  1,  34)  =  9.21 , p  <  .005.  Spatial  ability  was  a  significant 
covariate  for  Mission  5,  F(l,  34)  =  4.35,  p  =  .05,  qp  =  .1 1,  but  in  an  unexpected  direction.  Higher 
spatial  ability  was  associated  with  higher  workload  scores,  Pearson  r  =  .19. 

Discussion 

Wide  deployment  of  MAVs  at  the  platoon  level  will  mean  a  growing  need  to  train  many 
operators,  and  likely  involve  some  simulation-based  training.  The  current  research  explored 
training  measures  for  such  a  purpose.  In  particular,  the  present  findings  indicated  that  temporal 
measures  of  perfonnance  are  relatively  sensitive  and  could  provide  a  measure  for  training  to  a 
criterion  for  both  flight  skill  missions  and  reconnaissance  missions.  Completing  simulated 
training  missions  within  set  temporal  parameters  could  be  added  to  other  prerequisite 
performance  criteria  (such  as  lack  of  collisions  and  number  of  targets  detected)  gating  hands-on 
training  with  an  actual  vehicle. 

Besides  investigating  the  temporal  performance  measure,  we  also  examined  the  factors 
which  affected  why  some  input  device  configurations  support  faster  mission  completion  than 
others.  In  our  previous  research,  participants  completed  missions  more  quickly  with  a  GC  than 
with  a  mouse  (GC1  vs.  M  in  the  present  experiment).  In  the  current  experiment  we  attempted  to 
un-confound  the  factors  distinguishing  these  devices  to  detennine  which  factors  contributed  to 
that  effect.  One  difference  between  a  GC  device  and  a  point  and  click  device  such  as  a  M  (or  a 
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touch  screen),  is  that  the  point  and  click  device  requires  the  user  to  switch  visual  attention 
between  the  sensor  imagery  and  the  icons,  which  must  be  selected  to  control  vehicle  movement 
and  functions.  In  contrast,  the  GC  allows  focused  attention  on  the  sensory  image,  because  all 
controls  can  be  operated  by  feel  (without  the  need  for  visual  attention  on  the  controls 
themselves).  We  therefore  reconfigured  our  GC  in  condition  GC2  to  make  it  operate  identically 
to  our  M  condition,  except  for  the  factor  of  visual  attention.  Because  performance  failed  to  differ 
across  GC2  and  M,  our  results  indicated  that  visual  attention  was  not  the  source  of  the  difference 
between  GC1  and  M. 

In  our  original  research,  the  GC  and  M  differed  in  the  control  they  afforded  the  user  of 
both  maneuver  directions  and  speed.  The  GC  allowed  simultaneous  control  of  all  axes  of 
maneuver  and  user-control  of  speed.  In  contrast,  the  M  allowed  only  one  axis  of  maneuver  at  a 
time  and  speed  was  controlled  by  the  system.  To  tease  apart  the  contribution  of  these  factors,  in 
this  experiment  we  engineered  three  new  GC  conditions,  and  examined  the  effects  of  each  factor 
in  a  2  x  2  (movement  control  x  speed  control)  analysis.  The  results  indicated  that  movement 
control  significantly  affected  mission  completion  time  on  flight  skill  missions,  although  not  on 
reconnaissance  missions.  Conversely,  speed  control  affected  mission  completion  time  on 
reconnaissance  missions,  but  not  flight  skill  missions.  The  interpretation  of  this  pattern  is  fairly 
straight-forward  with  some  reflection  on  the  mission  requirements.  Flight  skill  missions  required 
route  following  and  maneuver  around  obstacles.  The  ability  to  move  with  a  curved  trajectory  or 
off  at  an  angle  while  still  moving  forward  would  be  particularly  useful  and  much  easier  to 
accomplish  with  multi-directional  control.  The  reconnaissance  missions  required  not  only  the 
finding  of  targets,  but  also  their  photographing.  The  ability  to  creep  ahead  at  a  very  slow  speed 
would  be  particularly  useful  when  trying  to  line  up  the  camera  focal  point  with  the  target  (which 
was  required).  Under  system-speed  control,  moving  faster  than  desired  was  likely  an  impediment 
to  getting  the  target  lined  up,  and  operators  may  have  had  to  stop  and  hover  to  get  the  picture.  In 
contrast,  with  user-controlled  speed,  operators  could  move  very  slowly  and  take  a  photo  without 
having  to  come  to  a  stop.  This  could  have  enabled  them  to  complete  reconnaissance  missions 
more  quickly. 

Interestingly,  the  results  of  this  experiment  are  completely  compatible  with  some  results 
of  our  original  experiment,  not  heretofore  mentioned.  In  that  experiment  (Durlach,  Neumann,  & 
Billings,  2008)  we  included  a  condition  in  which  participants  used  a  M,  but  had  different  on¬ 
screen  control  iconology  from  that  used  here.  That  control  scheme  allowed  both  user-control  of 
speed  and  multiple  directions  of  maneuver  simultaneously.  Mission  completion  times  for  flight 
skill  missions  for  that  condition  were  as  good  as  in  the  GC  condition  on  some  missions.  Thus,  in 
that  experiment,  we  improved  performance  with  the  M  by  adding  the  ability  to  maneuver  in 
multiple  directions  simultaneously  and  control  speed.  In  the  present  experiment,  we  undennined 
performance  with  the  GC  by  removing  the  ability  to  maneuver  in  multiple  directions 
simultaneously  or  control  speed. 

With  regard  to  workload  scores  for  this  experiment,  like  for  our  previous  experiments, 
participants  with  higher  VGE  generally  found  the  tasks  less  demanding,  regardless  of  condition. 
In  addition,  here,  participants  in  GC  1  (the  original  GC  condition)  often  reported  significantly 
lower  subjective  workload  than  participants  in  M.  Sporadically,  significantly  lower  workload 
was  also  reported  by  participants  with  multi-directional  control  (GC  1  and  GC3),  compared  with 
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those  with  only  uni-directional  control  (GC2  and  GC4).  It  is  not  clear  why  workload  reports  were 
more  sensitive  to  input  device  characteristics  in  this  experiment  than  in  our  prior  research. 
Previously,  this  difference  was  detected  only  once  (Durlach,  Neumann,  &  Billings,  2008,  Church 
Mission).  Systems  designers  may  sometimes  impose  constraints  on  operators’  freedom  of 
control  in  order  to  lessen  workload  or  the  training  requirements.  For  example,  the  t-MAV 
prototype  (Durlach,  2007)  limited  the  velocity  of  the  MAV  to  six  knots  in  manual  mode,  despite 
the  capability  of  the  system  to  fly  at  speeds  over  25  knots.  This  limitation  was  intended  to 
prevent  aerodynamic  instabilities,  which  could  occur  if  particular  patterns  of  maneuver 
commands  were  issued  when  the  MAV  was  traveling  at  faster  speeds.  By  limiting  velocity  in 
manual  mode,  the  need  to  train  operators  on  the  types  of  maneuvers  they  should  avoid  at  higher 
speeds  was  eliminated.  The  present  results  indicate  that  multi-directional  control  and  options  to 
move  at  slower  speeds  aid  perfonnance  without  adding  a  workload  or  training  burden. 

The  results  of  this  investigation  are  concordant  with  others  which  have  suggested  that 
matching  input  device  characteristics  to  task  characteristics  can  lead  to  performance  benefits 
(Jacob,  Sibert,  McFarlane,  &  Mullen,  1994;  Rogers,  Fisk,  McLaughlin,  &  Pak,  2005; 
McLaughlin,  Rogers,  &  Fisk,  2009).  Optimizing  this  matching  may  be  challenging,  however, 
when  the  same  input  device  is  required  for  accomplishing  various  tasks.  For  example,  in  the 
Durlach,  Neumann,  &  Billings  (2008)  condition  where  the  iconology  afforded  user-speed  control 
and  multi-directional  control  with  the  M,  flight  skill  performance  was  superior  to  the  standard  M 
condition  (the  same  as  that  used  in  the  present  research);  however,  target  detection  and 
photography  was  impaired.  We  inferred  this  was  due  to  that  configuration’s  lack  of  a  single 
discrete  command  to  hover.  With  hindsight,  it  seems  obvious  that  a  discrete  command  would  be 
best  for  this;  however,  in  practice,  it  may  be  difficult  to  anticipate  how  input  device 
characteristics  and  on-screen  display  characteristics  will  interact.  Although  matching  task  to 
input  device  is  critical,  research  is  still  needed  to  address  the  challenge  arising  from  using  one 
input  device  for  many  different  tasks,  which  is  a  more  realistic  situation  in  most  human- 
computer  interfaces. 

Finally,  in  the  present  experiment  we  failed  to  find  an  effect  of  focused  vs.  divided 
attention  on  performance.  This  should  not  be  taken  to  imply  that  attentional  factors  are  not 
important,  however.  Use  of  input  devices  does  require  attention,  and  the  match  between  device 
and  task  may  be  more  important  as  the  task  becomes  more  attentionally  demanding  for  the  user. 
McLaughlin,  Rogers,  and  Fisk  (2009)  found  not  only  that  task  perfonnance  was  influenced  by 
the  match  between  input  device  and  task  requirements,  but  also  that  older  participants  benefited 
more  than  younger.  This  benefit  increased  as  less  attentional  resources  were  available.  Hence, 
even  though  our  current  investigation  did  not  suggest  any  impact  of  attention  on  task 
performance,  other  research  implies  that  attentional  demands  do  actually  contribute  to 
performance  differences  among  input  devices. 

Implications  for  Instructional  Design  for  Teleoperation 

•  To  achieve  effective  standards-based  training  for  teleoperation,  simulation-based  training 
should  include  a  series  of  progressively  more  challenging  missions.  Each  step  should 
have  an  associated  performance  criterion  which  must  be  met  before  advancing  to  the  next 
more  challenging  level.  Missions  should  cover  the  entire  range  of  tasks  the  operator 
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would  be  expected  to  perform.  Rationale:  Unstructured  training  simulations  merely 
familiarize  students,  but  do  not  enforce  mastery  criteria  for  a  specific  set  of  skills.  We 
have  shown  that  by  structuring  training  and  providing  performance-based  feedback, 
students  can  learn  to  perform  complex  simulated  missions  relatively  quickly. 

•  In  designing  standards-based  simulation  training  for  teleoperation,  temporal  measures  are 
the  very  discriminating  performance  metrics  and  should  be  combined  with  required,  but 
less  sensitive  performance  metrics  to  assess  mastery.  Rationale:  During  the  course  of  our 
experiments  we  found  that  discrete  performance  measures  such  as  number  of  targets 
detected  or  number  of  collisions  were  relatively  insensitive  to  manipulation  of  the  input 
device.  They  may  represent  a  minimum  requirement  for  standards-based  training  (e.g.,  a 
mission  must  be  completed  without  any  collisions);  however,  to  discriminate  different 
degrees  of  mastery  above  and  beyond  these  essential  requirements,  mission  completion 
criterion  times  are  appropriate. 

•  The  possibility  that  experienced  video-game  players  may  perfonn  better  in  teleoperation 
should  be  further  investigated.  Rationale:  We  have  consistently  found  that  experienced 
video-game  players  master  teleoperation  in  simulation  more  quickly  than  nonplayers; 
however,  it  is  unknown  whether  this  generalizes  to  live  environments. 

Implications  for  the  Design  of  Teloperation  Input  Controls 

•  Matching  user  input  device  characteristics  to  required  task  characteristics  improves 
teleoperation  performance.  The  entire  gamut  of  tasks  to  be  performed  must  be  considered 
to  maximize  the  benefits.  Rationale:  We  found  that  different  aspects  of  input  device 
control  affected  different  tasks  differently.  The  ability  to  maneuver  in  multiple  directions 
at  one  time  affected  route  following,  but  not  target  photography.  Conversely,  the  ability 
to  control  speed  affected  target  photography  but  not  route  following. 

•  The  specific  way  in  which  an  input  device  capability  is  implemented  and  not  just  the 
capability  itself,  is  important,  because  the  implementation  may  affect  performance  on 
other  tasks.  Rationale:  We  found  (across  experiments)  that  the  ability  to  maneuver  in 
multiple  directions  at  a  time  facilitated  route  following.  Depending  on  how  it  was 
implemented,  however,  it  differently  affected  performance  on  other  tasks. 
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Appendix  A 


List  of  Acronyms 


ANCOVA 

Analysis  of  Covariance 

ARI 

Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

GC 

Game  controller 

GC1 

Game  controller,  1st  configuration 

GC2 

Game  controller,  2nd  configuration 

GC3 

Game  controller,  3rd  configuration 

GC4 

Game  controller,  4th  configuration 

M 

Mouse 

MAV 

Micro-unmanned  aerial  vehicle 

MOUT 

Military  operations  on  urban  terrain 

NASA  TLX 

NASA  Task  Load  Index 

ocu 

Operator  control  unit 

OTB 

One-SAF  Testbed 

SUAS 

Small  Unmanned  aerial  system(s) 

UAS(s) 

Unmanned  aerial  system(s) 

UAV 

Unmanned  aerial  vehicle 

VGE 

Video  game  experience 
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Appendix  B 


Demographics  Questionnaire 

1 .  Y ear  of  birth: _ 

2.  Gender: 

□  Male  □  Female 

3.  Have  you  graduated  from  high  school? 

□  Yes  EH  No 

4.  Which  hand  do  you  write  with? 

□  Left  □  Right 

5.  Is  your  vision  in  each  eye  correctable  to  20/20? 

□  Yes  EH  No 

6.  To  your  knowledge,  are  you  color  blind? 

EH  Yes  EH  No 

7.  Do  you  own  or  have  access  to  a  computer? 

EH  Yes  EH  No 


8.  If  yes,  how  often  do  you  use  a  computer? 

EH  Daily  EH  Several  times  a  week  EH  Occasionally 


□  Never 


9.  Estimate  how  many  hours  per  week  you  use  a  computer. 

EH  Never  EH  1-5  hrs  EH5-10hrs  EH  10-20  hrs  EH  20-30  hrs 
□  30-40  hrs  □  40  +  hrs 
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10.  How  do  you  rate  your  computer  skills? 

EH  Novice/Beginner  EH  Intermediate  EH  Expert 

1 1 .  Do  you  use  the  Internet? 

EH  Yes  EH  No 

12.  Do  you  have  any  previous  remote  control  (R/C)  experience  (including  cars,  boats,  etc.)? 

EH  Yes  EH  No 


13.  Do  you  own  or  use  a  video  game  system? 
EH  Yes  EH  No 


14.  How  would  you  rate  your  video  game  skills? 
EH  Novice/Beginner  EH  Intermediate 


EH  Expert 


15.  How  often  do  you  play  video  games? 

EH  Daily  EH  Several  times  a  week 


EH  Occasionally  EH  Never 


16.  Estimate  how  many  hours  per  week  you  play  video  games. 

EH  Never  EH  1-5  hrs  EH  5-10  hrs  EH  10-20  hrs  EH  20-30  hrs 
□  30-40  hrs  □  40  +  hrs 
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Appendix  C 


Training  Evaluation  Worksheet 


To  ensure  that  you  have  a  basic  grasp  of  the  MAV  pilot  interface  and  the  available  flight  commands, 
please  complete  the  following  exercise. 

Each  of  the  critical  features  of  the  user  interface  is  labeled  above  with  letters  A  -  N.  Every  letter  must  be 
used,  so  choose  the  best  answer.  Enter  the  corresponding  letter  in  the  blank  following  each  of  the  item 
descriptions  below: 

Altimeter  _  Camera  Selection  Icons  _ 

Mission  Timer  _  Altitude  Control  _ 

MAV  Rotational  Icons  _  Take-off  and  Land  Icon  _ 

Heading  Tape  _  Halt  Movement  Icon  _ 

Satellite  Map  View  _  MAV  Location  on  Map  _ 

Current  MAV  Altitude  _  Ground  Level  Indicator  _ 

Forward  Camera  Image  _  Air  Speed  Indicator  _ 
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Appendix  D 


Experimenter’s  Script 


Army  Research  Institute 
Intuitive  Means  of  Robotic  Control 
Testbed 


Experimenter’s  Guide 


INSTITUTE  for 
SIMULATION 
Q TRAINING 


Exp  3  Follow-up  MAV  Study  -  1ST 
Summer/Fall  2008 


(Read  with  participant) 
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Introduction  to  the  Micro-Unmanned  Aerial  Vehicle  Study 


The  U.S.  Army  is  undergoing  a  major  transformation.  One  element  of  the  transformation  is  the 
introduction  of  a  new  class  of  military  platforms  known  as  unmanned  air  and  ground  vehicles 
( called  UAVs  and  UGVs).  A  major  benefit  of  these  unmanned  vehicles  is  that  they  can  perform 
reconnaissance  missions  and  survey  areas  contaminated  with  radiological,  chemical,  or 
biological  agents  without  risk  to  human  life.  They  can  also  survey  the  battlefield  and  provide 
real-time  video  feedback. 

We  are  investigating  the  design  of  operator  control  systems  for  micro-unmanned  aerial  vehicles 
that  can  perform  these  kinds  of  reconnaissance  missions.  In  addition,  we  are  investigating 
operator  training  requirements.  In  this  experiment,  you  will  be  trained  on  how  to  fly  a  simulated 
micro-UAV  (MAV),  and  then  you  will  complete  a  set  of  missions  that  will  test  your  ability  to 
maneuver  the  MAV  and  locate  various  targets.  After  each  mission,  you  will  be  given  a  short 
questionnaire  that  asks  you  to  rate  certain  aspects  of  the  task  you  performed. 

We  have  allotted  approximately  3  hours  for  this  experiment,  although  most  people  finish 
in  less  time.  No  previous  flight  experience  is  necessary  to  participate  in  this  study. 


Confidentiality 

Your  identity  will  be  kept  confidential  to  the  extent  provided  by  law.  Your  information  will  be 
assigned  a  code  number.  The  list  connecting  your  name  to  this  number  will  be  kept  in  an 
electronic  file.  When  the  study  is  completed  and  the  data  have  been  analyzed,  the  list  will  be 
destroyed.  You  name  will  not  be  used  in  any  report. 

If  you  are  prepared  to  participate  in  this  experiment,  please  read  and  sign  the  Consent  Form 
and  Voluntary  Agreement.  Please  also  indicate  on  that  form  your  preferred  method  of 
compensation.  We  offer  cash  payment  or  course  credit.  You  will  receive  compensation  per  half 
hour  you  participate  in  this  experiment. 


1)  Give  each  participant  the  informed  consent  form  to  read  and  sign  before 
disclosing  any  details  about  the  study. 

2)  Once  the  informed  consent  is  signed,  the  participant  should  complete  the 

demographics  questionnaire. 

3)  After  this  is  finished,  the  experimenter  should  administer  the  written  ETS  paper- 
based  Hidden  Patterns  spatial  test. 


All  forms  such  as  the  informed  consent  should  remain  on  file. 


After  participants  complete  the  Hidden  Patterns  spatial  test,  tell  them  to  begin  by  reading  page  3 
in  their  Participant  Manuals. 

(Participant  will  read  the  following  by  himself/herself): 
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Overall  Description  of  the  MAV  Simulation 


You  will  be  working  with  a  simulation  for  flying  a  micro-unmanned  aerial  vehicle.  The  micro-UAV 
itself  will  be  referred  to  from  now  on  as  “the  MAV.”  This  is  not  a  fixed-wing  aircraft  like  most 
airplanes  are;  rather,  it  is  a  small  rotary  craft  with  an  internal  fan  and  duct  design  (see  prototype 
photos  below).  An  operator  controls  the  MAV  using  a  laptop  computer  equipped  with  an  input 
device  such  as  a  mouse  or  game  controller.  This  interface  is  referred  to  as  the  OCU  -  Operator 
Control  Unit.  This  is  a  dismounted  control  unit,  as  it  is  envisioned  that  a  dismounted  soldier  (on 
foot  rather  than  in  a  vehicle)  will  be  controlling  the  MAV. 


Prototype  Micro  UAVs  from  Honeywell  and  Allied  Aerospace 


Introduction  to  the  OCU  and  MAV  Camera  System 

The  MAV  is  equipped  with  a  dual  camera  system.  When  the  vehicle  flies  through  the  simulated 
environment  you  will  be  able  to  view  video  images  sent  back  to  the  OCU.  You  will  be  instructed 
on  how  to  operate  the  cameras  as  well  as  how  to  use  the  OCU  interface  and  controllers  to  pilot 
the  MAV.  You  will  also  have  an  opportunity  to  practice  some  manual  flight/piloting  techniques 
before  beginning  the  actual  experiment.  After  basic  instruction,  a  training  session  will  take 
place;  then  you  will  move  on  to  the  assigned  pilot  mission  tasks  where  performance  data  will  be 
recorded.  Be  sure  that  you  understand  the  objective  of  each  mission  before  starting  a  trial.  The 
experimenter  is  available  to  answer  your  questions  before  you  begin  each  task,  so  please  ask 
for  help  if  you  are  unsure  of  any  requirements.  Unless  instructed  otherwise,  it  is  important  that 
you  complete  each  task  as  quickly  and  efficiently  as  possible. 

At  the  end  of  the  training  session  and  at  the  end  of  each  mission  you  will  complete  a  short 
computer-based  questionnaire.  In  the  first  section  you  will  rate  different  aspects  of  the  task  you 
performed;  then  you  will  be  asked  to  choose  between  a  pair  of  items  that  relate  to  your 
performance.  You  must  choose  one  and  only  one  item  for  each  pair.  Then  you  will  proceed  to 
the  next  mission.  Please  let  the  experimenter  know  when  you  are  done  reading  this  section. 
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Training  Session  and  Exercises 

Experimenter:  When  the  participant  reaches  this  point  in  the  handout,  explain  that  a 
short  training  session  will  be  administered.  You  should  have  a  good  understanding  of  the 
MAV  flight  characteristics  to  run  this  study. 

(Begin  this  and  every  exercise  by  reviewing  the  mission  objectives  with  the  participant.) 


At  this  time  load  the  correct  condition  properties  file  (.opf).  To  do  this  click  FILE  - 
Properties,  then  FILE  -  Open,  and  select  the  property  (.opf)  file  from  the  list.  Then  click 
“apply”  to  begin  the  simulation. 


Condition 

File  name 

1 

Focused,  multiple,  user-defined  speed  (GC) 

exp3 cond1  .opf 

2 

Divided,  single,  system-defined  speed  (MOUSE) 

exp3 cond2.opf 

3 

Focused,  single,  system-defined  speed  (GC) 

exp3  cond3.opf 

4 

Focused,  multiple,  system-defined  speed  (GC) 

exp3  cond4.opf 

5 

Focused,  single,  user-defined  speed  (GC) 

exp3 cond5.opf 

(Read  the  following  with  participant): 

Training  Session 

The  goal  of  training  is  to  familiarize  you  with  the  flight  characteristics  of  the  MAV  and  to  give  you 
an  opportunity  to  practice  piloting  the  MAV.  We  will  begin  by  reviewing  the  features  of  the  OCU 
and  then  proceed  to  a  series  of  practice  exercises.  I  will  facilitate  this  training  session  and 
provide  instruction  on  how  to  complete  the  assigned  tasks. 

OCU  Layout  and  On-Screen  Controls 

Below  you  will  see  a  sample  layout  of  the  OCU.  On  the  left  side  of  the  screen  is  the  video 
sensor  imagery  (camera  views),  and  in  the  upper-right  side  you  will  find  an  overhead  map  view 
of  the  terrain  database.  Just  below  the  map  view  there  is  a  control  pad  that  is  used  in  issuing 
flight  commands  to  the  MAV.  The  control  pad  will  be  examined  in  greater  detail  throughout 
training.  There  is  an  altimeter  along  the  left  border  area  of  the  OCU  display  as  well  as  a  task  bar 
along  the  top  that  contains  various  icons.  We  will  go  over  how  to  use  all  relevant  gauges  and 
icons  during  training. 


Please  review  the  game  controller  buttons: 


(If  mouse  condition,  then  continue 

with  training) 
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Understanding  the  Task  Bar  Icons 


The  upper  task  bar  (shown  below)  includes  the  take-off  (and  landing)  and  camera  control 
buttons.  There  is  also  a  mission  timer  located  on  the  far  right  of  this  task  bar.  For  this  study, 
you  will  need  to  know  how  to  use  the  take-off  icon  and  the  camera  control  icons. 


Take-off  &  Landing  icon 

0 

You  can  take  off  and  land  by 

[FOR  MOUSE1  clicking  on  the  task  bar  icon  for  take-off  and  landing,  or  by 

[FOR  GAME  CONTROLLER)  pressing  button  (10)  to  activate  the  task  bar  icon  for  take¬ 
off  and  landing. 

Once  the  take-off  button  is  pressed  the  MAV  will  automatically  climb  to  an  altitude  of 
approximately  60  feet  above  the  ground  level.  You  will  not  be  able  to  enter  any  flight 
commands  to  the  MAV  until  you  see  a  red  stop  sign  icon  illuminate  (meaning  that  you  have 
reached  take-off  altitude). 

You  may  now  execute  the  take-off  command. 

Pressing  the  take-off  button  again  will  execute  the  land  command.  You  may  land  the  MAV  now. 


Mission  Mode  Icons  (For  Experimenter  Use  ONLY — Do  NOT  share  with  participant): 


These  mode  buttons  switch  between  control  modes. 


•  Mission  Mode  -  (1st  icon)  causes  the  MAV  to  automatically  navigate  through  pre¬ 
programmed  waypoints,  (experimenter  only  does  this) 

•  Go-To  Mode  -  (2nd  icon)  allows  the  operator  to  select  a  location  on  the  map  for  the 
MAV  to  travel  to  by  clicking  on  a  point  in  the  map  frame.  We  will  not  use  Go-To  in  this 
study. 

•  Manual  Mode  -  (3rd  icon)  allows  the  participant  to  manually  pilot  the  MAV  by  using 
the  mouse  or  joystick  controller.  (As  per  the  experiment) 

•  Idle  Mode  -  (4th  icon)  stops  the  MAV. 
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Activating  Camera  Views  and  taking  Snapshot  photos 


These  camera  buttons  allow  the  operator  to  switch  between  the  two  available  camera  images. 
On  the  OCU,  camera  image  #1  is  the  view  from  the  MAV’s  forward  camera,  and  camera  image 
#2  displays  the  view  from  the  downward  camera.  In  this  experiment,  one  camera  view  will  be 
displayed  at  a  time  so  the  operator  will  need  to  switch  views  in  order  to  activate  a  camera  for 
taking  snapshots. 

The  active  camera  image  will  always  have  a  blue  border  on  the  top  of  the  view  frame  as  well  as 
an  ( )  overlay  in  the  center  of  the  image.  The  corresponding  icon  on  the  task  bar  will  also 
illuminate.  To  change  the  active  camera  view,  you  must  use  the  [MOUSE/GAME 
CONTROLLER]  to  activate  the  group  of  icons  on  the  task  bar  and  select  the  camera  you  want. 


To  activate  a  camera  view: 

FOR  MOLISE:  This  is  done  by  using  the  mouse  to  click  on  the  desired  camera  view  in  the  task 
bar.  I  will  demonstrate  this  feature  now. 

FOR  GAME  CONTROLLER:  This  is  done  by  pressing  and  holding  button  (2)  while  you  scroll 
through  the  available  camera  views  with  the  directional  pad.  I  will  demonstrate  this  feature  now. 


If  not  already  airborne,  try  taking  off  and  switching  the  camera  view. 

FOR  MOLISE:  Snapshots  of  targets  can  now  be  taken  by  using  the  mouse  to  right-click  on  the 
frame  of  the  active  camera  view. 

FOR  GAME  CONTROLLER:  Snapshots  of  targets  can  now  be  taken  using  button  (9)  on  the 
game  controller. 


(Continue  going  over  the  manual  with  the  participant): 
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Main  Window  Components 

Altimeter  -  (See  vertical  bar  on  the  left  side  of  this  page) 

The  ruler-like  markings  on  the  altimeter  display  the  altitude  of  the  MAV  in  feet  above  sea  level.  If 
the  experimenter  has  activated  the  altitude  alarm  system,  flying  above  or  below  the  specified 
altitudes  will  result  in  a  visual  alarm  on  the  altimeter.  The  experimenter  will  notify  you  if  an  alarm 
has  been  set. 


<  The  white  triangle  cut-out  points  to  the  current  altitude  of  the  MAV.  In  this  case,  the  MAV  is 
approximately  82  feet  above  sea  level. 


The  light  brown  column  at  the  lower  end  of  the  bar  marks  the  altitude  of  the  nearest  surface 
below  the  MAV  (this  is  the  current  ground  level). 

Note!~  In  the  current  example,  the  MAV  is  approximately  82  feet  above  sea  level,  but  the 
ground  level  is  approximately  22  feet.  This  means  the  MAV  is  only  60  feet  above  ground! 


Input  Control  Pad  Game  Controller: 


You  will  issue  commands  to  the  MAV  by  using  the  game  controller,  and  you  will  receive 
feedback  from  the  on-screen  input  control  pad. 

The  four  arrow  icons  will  illuminate  when  forward,  backward,  (strafe)  left,  and  (strafe)  right 
commands  are  issued  to  the  MAV.  The  curved  arrow  icons  in  the  upper  corners  will  illuminate 
when  rotate  commands  are  given.  The  lower-corner  icons  will  illuminate  when  you  issue  the 
ascend  or  descend  command  to  the  MAV.  The  middle  X  icon  lights  up  if  you  give  a  stop/hover 
command  to  the  MAV. 

The  control  pad  provides  feedback  in  this  way  to  the  operator  that  a  command  has  been  issued 
and  is  being  executed.  As  discussed,  arrow  icons  on  the  control  pad  will  illuminate  when  that 
command  has  been  entered  (e.g.,  when  you  push  forward  on  the  game  controller,  the  forward 
arrow  will  illuminate).  The  brighter  the  icon  gets,  the  faster  you  are  traveling.  Your  airspeed  is 
also  shown  here  in  knots.  The  max  speed  is  6  knots. 
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We  will  now  go  over  how  to  use  the  game  controller  for  all  of  the  tasks  required  during  this 
experiment. 

Input  Control  Pad  Mouse: 


You  will  issue  commands  to  the  MAV  by  using  the  mouse,  and  you  will  receive  feedback  from 
the  on-screen  input  control  pad.  This  input  control  pad  lets  you  control  the  position  of  the  MAV 
manually.  For  this  display,  nine  buttons  are  used  as  the  interface  to  the  MAV. 

The  four  arrow  icons  move  the  MAV  forwards,  backwards,  (strafe)  left,  and  (strafe)  right.  The 
curved  arrow  icons  in  the  upper  corners  rotate  the  MAV  left  and  right.  The  lower  corner  icons 
move  the  MAV  up  and  down.  The  middle  X  icon  stops  the  MAV. 

You  must  use  the  mouse  to  click  on  the  control  display  in  order  to  activate  the  MAV.  Arrow  icons 
on  the  discrete  control  pad  will  illuminate  when  a  command  has  been  entered  (e.g.,  when  you 
click  on  the  forward  arrow  on  the  control  pad,  the  forward  arrow  will  illuminate).  The  brighter  the 
icon  gets,  the  faster  you  are  traveling.  Your  airspeed  is  also  shown  here  in  knots.  The  max 
speed  is  6  knots. 


We  will  now  go  over  how  to  use  the  mouse  for  all  of  the  tasks  required  during  this  experiment. 


Controlling  MAV  movement  with  the  Mouse  (condition  2): 

The  left  mouse  button  is  responsible  for  clicking  and  activating  icons  on  the  task  bar  as  well  as 
the  directional  arrows  on  the  control  pad  to  direct  MAV  movement,  actions,  and  camera  views. 

The  right  mouse  button  is  responsible  for  taking  snapshots.  The  scroll  wheel  will  not  be  used 
during  this  simulation.  At  this  time,  you  may  take  off  and  practice  piloting  the  MAV  for  1-2 
minutes. 


Controlling  MAV  movement  with  the  Game  Controller  (condition  1): 

The  left  thumb  stick  controls  movement  forward,  backward,  and  sideways  (or  at  angles),  but 
the  MAV  heading  never  changes  when  the  left  thumb  stick  is  used.  Pushing  up  on  this  thumb 
stick  moves  the  MAV  forward.  Pulling  down  on  the  stick  moves  the  MAV  backwards.  Moving  the 
stick  from  side  to  side  moves  the  MAV  right  or  left  without  altering  heading  (moving  laterally). 
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The  right  thumb  stick  controls  altitude  and  headinq/rotation.  Pushing  up  on  the  stick  increases 
the  MAV’s  altitude,  and  pulling  down  decreases  altitude.  Moving  the  right  thumb  stick  from  side 
to  side  rotates  the  MAV  in  place. 

It  is  possible  (and  expected)  to  use  both  sticks  simultaneously  to  rotate,  change  altitude,  and 
move  at  the  same  time.  Vehicle  speed  is  graded,  meaning  that  barely  pressing  the  thumb  stick 
will  cause  the  MAV  to  move  more  slowly  than  pressing  it  all  the  way  down.  Releasing  the  thumb 
sticks  will  slow  and  stop  the  MAV,  causing  it  to  hover  in  mid-air. 

At  this  time,  you  may  take  off  and  practice  manipulating  the  thumb  sticks  for  1-2  minutes. 


Controlling  MAV  movement  with  the  Game  Controller  (condition  3): 

The  left  thumb  stick  controls  movement  forward,  backward,  and  sideways  (or  at  angles),  but 
the  MAV  heading  never  changes  when  the  left  thumb  stick  is  used.  Tapping  up  with  this  thumb 
stick  moves  the  MAV  forward.  Tapping  down  with  the  stick  moves  the  MAV  backwards.  Tapping 
the  stick  from  side  to  side  moves  the  MAV  right  or  left  without  altering  heading  (moving 
laterally).  You  should  not  hold  the  thumb  stick  down  when  sending  a  command — always  press 
and  release  to  issue  a  command. 

The  right  thumb  stick  controls  altitude  and  headinq/rotation.  Tapping  up  with  the  stick 
increases  the  MAV’s  altitude,  and  tapping  down  decreases  altitude.  Moving  the  right  thumb  stick 
from  side  to  side  and  releasing  rotates  the  MAV  in  place.  You  should  not  hold  the  thumb  stick 
down  when  sending  a  command — always  press  and  release  to  issue  a  command. 

The  stop/hover  command  is  given  by  pressing  in  and  releasing  either  thumb  stick.  I  will 
demonstrate  this  now. 

You  are  only  able  to  move  in  ONE  direction  at  a  time.  A  new  command  will  cancel  the  previous 
command. 

At  this  time,  you  may  take  off  and  practice  manipulating  the  thumb  sticks  for  1-2  minutes. 


Controlling  MAV  movement  with  the  Game  Controller  (condition  4): 

The  left  thumb  stick  controls  movement  forward,  backward,  and  sideways  (or  at  angles),  but 
the  MAV  heading  never  changes  when  the  left  thumb  stick  is  used.  Tapping  up  with  this  thumb 
stick  moves  the  MAV  forward.  Tapping  down  with  the  stick  moves  the  MAV  backwards.  Tapping 
the  stick  from  side  to  side  moves  the  MAV  right  or  left  without  altering  heading  (moving 
laterally).  You  should  not  hold  the  thumb  stick  down  when  sending  a  command — always  press 
and  release  to  issue  a  command. 

The  right  thumb  stick  controls  altitude  and  headinq/rotation.  Tapping  up  with  the  stick 
increases  the  MAV’s  altitude,  and  tapping  down  decreases  altitude.  Tapping  the  right  thumb 
stick  from  side  to  side  rotates  the  MAV  in  place.  You  should  not  hold  the  thumb  stick  down  when 
sending  a  command — always  press  and  release  to  issue  a  command. 
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The  stop/hover  command  is  given  by  pressing  in  and  releasing  either  thumb  stick.  I  will 
demonstrate  this  now. 

It  is  possible  (and  expected)  to  use  both  sticks  simultaneously  to  rotate,  change  altitude,  and 
move  at  the  same  time.  To  do  this,  you  must  press  and  release  the  controls  at  the  same  time. 

Now,  you  may  take  off  and  practice  manipulating  the  thumb  sticks  for  1-2  minutes. 


Controlling  MAV  movement  with  the  Game  Controller  (condition  5): 

The  left  thumb  stick  controls  movement  forward,  backward,  and  sideways  (or  at  angles),  but 
the  MAV  heading  never  changes  when  the  left  thumb  stick  is  used.  Pushing  up  on  this  thumb 
stick  moves  the  MAV  forward.  Pulling  down  on  the  stick  moves  the  MAV  backwards.  Moving  the 
stick  from  side  to  side  moves  the  MAV  right  or  left  without  altering  heading  (moving  laterally). 

The  right  thumb  stick  controls  altitude  and  headinq/rotation.  Pushing  up  on  the  stick  increases 
the  MAV’s  altitude,  and  pulling  down  decreases  altitude.  Moving  the  right  thumb  stick  from  side 
to  side  rotates  the  MAV  in  place. 

You  are  only  able  to  move  in  ONE  direction  at  a  time.  A  new  command  will  cancel  the  previous 
command.  Vehicle  speed  is  graded,  meaning  that  barely  pressing  the  thumb  stick  will  cause  the 
MAV  to  move  more  slowly  than  pressing  it  all  the  way.  Releasing  the  thumb  sticks  will  slow  and 
stop  the  MAV,  causing  it  to  hover  in  mid-air. 

At  this  time,  you  may  take  off  and  practice  manipulating  the  thumb  sticks  for  1-2  minutes. 


Game  Controller  Configuration: 


Left  thumb  stick— lateral 
movement,  forward  & 
backward  movement 


Release  the  thumb  sticks  and  the 
MAV  will  stop/hover. 


Right  thumb  stick- 
rotate,  ascend, & 
descend 


Button  2—  hold  and  scroll  through 
camera  views  with  directional  pad; 
release  on  the  one  you  want 
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Heading  Tape 


Note  that  each  camera  view  window  has  a  heading  tape  located  along  the  top  edge  of  the 
frame.  This  number  indicates  the  current  heading  of  the  vehicle  (based  on  360  degrees)  with 
regard  to  that  camera  image.  Because  the  cameras  have  been  locked  in  place  for  this 
experiment,  you  can  assume  that  the  forward  camera  view  heading  is  the  same  as  the  MAV 
heading. 


Heading  tape 


If  the  heading  tape  =  0° 


Important  Note!!  The  downward  camera 
view  is  locked  at  90  degrees  from  horizontal, 
which  is  essentially  straight  down.  However, 
when  the  MAV  is  in  motion  the  vehicle  tilts  in 
a  similar  manner  to  a  moving  helicopter. 

This  tilting  will  cause  the  downward  camera 
to  point  slightly  backwards  when  moving. 
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Practice  Time:  Now  that  you  have  learned  all  the  functions  of  the  operator  control  unit  (OCU) 
and  the  flight  controls,  we  will  complete  a  series  of  practice  exercises  beginning  on  the  next 
page. 

Practice  Exercises 


These  exercises  will  give  you  a  chance  to  practice  the  various  tasks  required  to  complete  the 
missions  in  this  study. 

For  each  of  these  simulated  exercises,  you  will  be  controlling  the  MAV  from  a  C2  (command- 
control)  vehicle  (which  looks  like  a  Hummer).  Your  C2  vehicle  is  marked  on  the  satellite  map 
with  a  blue  icon. 


=  C2  vehicle 


(Continue  going  over  the  manual  with  the  participant): 


FOR  MOUSE:  Warm  Up:  Load  the  correct  properties  file — exp3  cond2.opf .  No 
mission  file  or  OneSAF  scenario  is  required  for  this  exercise. 


The  experimenter  will  call  out  movement  commands  one  by  one,  and  you  need  to  move  the 
MAV  accordingly.  You  will  notice  that  inertia  comes  into  play  when  trying  to  stop  the  MAV,  so 
you  will  need  to  learn  to  estimate  things  like  stopping  distances  and  rotational  velocity  carry¬ 
over. 


1 )  Use  the  mouse  to  click  on  the  up  and  down  arrows  located  on  the  bottom  left  and  right  of 
the  control  pad  to  make  the  MAV  ascend  and  descend. 

2)  Click  the  curved  arrows  to  rotate  the  MAV  left  and  right. 

3)  Click  the  up  and  down  arrows  to  make  the  MAV  fly  forward  and  backwards. 

4)  Click  the  left  and  right  arrows  to  move  the  MAV  laterally.  These  arrows  allow  you  to 
strafe  left  and  right.  Note  that  the  heading  only  changes  when  you  rotate  the  MAV  with 
the  curved  arrows. 

5)  Click  the  X  to  halt  movement  of  the  MAV.  Note  that  the  MAV  will  not  stop  immediately; 
instead,  it  will  decelerate  and  then  come  to  a  halt. 

6)  Activate  camera  2  (by  clicking  the  camera  icons  on  the  task  bar),  and  then  activate 
camera  1 .  Now  switch  back  to  camera  2  and  take  a  snapshot  (by  right-clicking  on  the 
camera  frame). 

7)  Land  the  MAV. 


Next  you  will  complete  a  series  of  timed  practice  exercises.  The  experimenter  will 
observe  these  exercises  and  determine  if  you  have  met  the  time  requirement  before 
allowing  you  to  proceed  to  the  next  exercise.  You  have  5  attempts  to  successfully 
complete  each  of  the  practice  exercises.  Failure  to  complete  within  5  attempts  will  result 
in  a  dismissal  from  the  experiment.  If  this  happens,  you  will  be  compensated  for  the  time 
vou  DarticiDated. 
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FOR  GAME  CONTROLLER:  Warm  Up:  Load  the  correct  properties  file —  No  mission  file 
or  OneSAF  scenario  is  required  for  this  exercise. 


The  experimenter  will  call  out  movement  commands  one  by  one,  and  you  need  to  move  the 
MAV  accordingly.  You  will  notice  that  inertia  comes  into  play  when  trying  to  stop  the  MAV,  so 
you  will  need  to  learn  to  estimate  things  like  stopping  distances  and  rotational  velocity  carry¬ 
over. 


1 )  Move  the  right  thumb  stick  up  and  down  to  make  the  MAV  ascend  and  descend. 

2)  Move  the  right  thumb  stick  side  to  side  to  rotate  the  MAV. 

3)  Move  the  left  thumb  stick  up  and  down  to  make  the  MAV  fly  forward  and  backwards. 

4)  Move  the  left  thumb  stick  side  to  side  to  move  the  MAV  laterally.  Note  that  the  heading 
only  changes  when  you  rotate  the  MAV  with  the  right  thumb  stick. 

5)  Activate  camera  2,  and  then  activate  camera  1 .  Now  switch  back  to  camera  2  and  take  a 
snapshot. 

6)  Land  the  MAV. 


Next  you  will  complete  a  series  of  timed  practice  exercises.  The  experimenter  will 
observe  these  exercises  and  determine  if  you  have  met  the  time  requirement  before 
allowing  you  to  proceed  to  the  next  exercise.  You  have  5  attempts  to  successfully 
complete  each  of  the  practice  exercises.  Failure  to  complete  within  5  attempts  will  result 
in  a  dismissal  from  the  experiment.  If  this  happens,  you  will  be  compensated  for  the  time 
vou  participated. 
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FOR  MOUSE  &  GAME  CONTROLLER:  Practice  Exercises 


Practice  Exercise  1 :  Load  the  correct  properties  file  for  the  condition —  No  mission  file  or 
OneSAF  OTB  scenario  is  required  for  this  exercise. 

We  will  go  over  the  practice  exercise  requirements  before  you  begin: 

1 )  Click  OK  (button  10  on  game  controller)  to  start  the  simulation  and  timer. 

2)  Execute  the  Take-off  command. 

3)  When  the  Red  Stop  icon  illuminates,  execute  the  Land  command. 

4)  This  must  be  completed  in  30  seconds  (:30)  or  less. 

Do  you  have  any  questions  about  the  practice  exercise? 


Practice  Exercise  2:  Load  the  correct  properties  file  for  the  condition —  No  mission  file  or 
OneSAF  OTB  scenario  is  required  for  this  exercise. 

We  will  go  over  the  practice  exercise  requirements  before  you  begin: 

1 )  Click  OK  (button  10  on  game  controller)  to  start  the  simulation  and  timer. 

2)  Execute  the  Take-off  command. 

3)  At  or  before  the  completion  of  take-off,  activate  the  view  window  for  camera  1  (forward 
view). 

4)  Take  a  snapshot  with  camera  1 . 

5)  Activate  the  view  window  for  camera  2  (down  view). 

6)  Take  a  snapshot  with  camera  2. 

7)  Execute  the  Land  command. 

8)  This  must  be  completed  in  40  seconds  (:40)  or  less. 

Do  you  have  any  questions  about  the  practice  exercise? 
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Practice  Exercise  3:  Load  the  correct  properties  file  for  the  condition —  No  mission  file  or 
OneSAF  OTB  scenario  is  required  for  this  exercise.  Set  the  upper  altitude  alarm  at  150  feet 
and  activate  it. 

We  will  go  over  the  practice  exercise  requirements  before  you  begin: 

1 )  The  upper  altitude  alarm  will  be  set  at  1 50  feet  and  activated. 

2)  Click  OK  (button  10  on  game  controller)  to  start  the  simulation  and  timer. 

3)  Execute  the  Take-off  command. 

4)  Ascend  to  a  little  above  150  feet,  and  you  will  trigger  the  alarm  automatically. 

5)  Immediately  descend  to  50  feet  or  below  without  hitting  the  ground. 

6)  Ascend  back  up  to  100  feet  but  less  than  150  feet. 

7)  Rotate  the  MAV  360-degrees  without  dropping  below  100  feet.  It  is  required  that  the 
heading  tape  shows  the  number  “0”  after  completing  1  rotation  with  the  MAV.  The  “0” 
must  remain  in  the  forward  camera  view  window  before  landing  (it  does  not  need  to  be 
centered,  just  viewable  on  the  screen). 

8)  Execute  the  Land  command. 

9)  This  exercise  must  be  completed  in  1  minute  35  seconds  (1 :35)  or  less. 


Do  you  have  any  questions  about  the  practice  exercise? 


Practice  Exercise  4:  rapid  command  execution:  Load  the  correct  properties  file  for  the 
condition —  No  mission  file  or  OneSAF  scenario  is  required  for  this  exercise. 

After  take-off  and  as  soon  as  the  Red  Stop  icon  illuminates,  you  will  immediately  begin  to  hear  a 
series  of  flight  commands.  Commands  will  be  given  as  fast  as  you  can  correctly  comply.  Once 
the  correct  feedback  is  observed  from  the  OCU,  [the  experimenter]  will  give  you  a  new 
command. 

Note:  It  is  not  important  that  the  MAV  travels  any  considerable  distance.  The  purpose  of  this 
exercise  is  to  allow  you  to  learn  the  mapping  of  all  buttons  and  icons  and  their  corresponding 
functions.  The  experimenter  is  looking  mainly  for  the  correct  feedback  from  the  OCU  control  pad 
located  in  the  lower  right  of  the  display. 

We  will  go  over  the  practice  exercise  requirements  before  you  begin: 


Rapid  command  execution  -  Part  A  Participant 

1)  Click  OK  (button  10)  to  start  the  simulation  and  timer. 

2)  Execute  Take-off. 

3)  The  first  series  of  commands  after  take-off  will  be:  9  commands 

4)  This  exercise  must  be  completed  in  1  minute  5  seconds  (1:05)  or  less. 
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Do  you  have  any  questions  about  the  practice  exercise? 


Rapid  command  execution  -  Part  A  Experimenter 

1 )  The  first  series  of  commands  after  take-off  should  be:  Ascend,  descend,  fly  forward, 
fly  (strafe)  right,  fly  (strafe)  left,  fly  backwards,  rotate  right,  rotate  left,  land. 

2)  This  exercise  must  be  completed  in  1  minute  5  seconds  (1 :05)  or  less. 

3)  Note  the  time  to  complete  and  reload  the  properties  file  (this  resets  timer). 


The  experimenter  will  now  reload  the  scenario. 


Rapid  command  execution  -  Part  B  Participant 

1)  Click  OK  (button  10)  to  start  the  simulation  and  timer. 

2)  Execute  Take-off. 

3)  The  first  series  of  commands  after  take-off  will  be:  14  commands 

4)  This  exercise  must  be  completed  in  1  minute  25  seconds  (1 :25)  or  less. 

Do  you  have  any  questions  about  the  practice  exercise? 


Rapid  command  execution  -  Part  B  Experimenter 

1 )  The  second  series  of  commands  after  take-off  should  be:  fly  forward,  fly  backwards, 
rotate  right,  fly  (strafe)  right,  fly  (strafe)  left,  activate  camera  2,  take  snapshot 
with  camera  2,  ascend,  descend,  hover,  activate  camera  1,  take  snapshot  with 
camera  1,  fly  forward,  land. 

2)  This  exercise  must  be  completed  in  1  minute  25  seconds  (1 :25)  or  less. 

3)  Note  the  time  to  complete. 


The  next  two  exercises  involve  flying  the  MAV  over  longer  distances  and  following  pre¬ 
determined  mission  parameters.  These  will  be  similar  to  the  missions  you  will  complete 
during  the  remainder  of  the  experiment. 
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Practice  Exercise  5:  Load  the  correct  properties  file  for  the  condition  (with  racetrack  poles,  see 
below) —  Then  load  the  mission  file:  Racetrack.ovm.  No  OneSAF  scenario  is  required  for  this 
exercise.  Run  the  mission  autonomously  with  waypoints  visible  and  point  out  the  Landing  Zone 
(LZ)  on  the  (H)  building. 

In  this  exercise  you  will  pilot  the  MAV  around  the  main  roadway.  Waypoints  will  be  visible  in  the 
overhead  map  view  window  so  that  you  can  refer  to  them  if  you  need  to  look  at  the  route. 

We  will  go  over  the  practice  exercise  requirements  before  you  begin: 

1)  The  experimenter  will  load  and  run  this  mission  autonomously  and  will  point  out  the 
Landing  Zone  (LZ)  on  the  (H)  building. 

2)  After  the  autonomous  mission  finishes,  the  simulation  will  be  reset. 

3)  You  must  now  pilot  the  MAV  around  the  gray  pathway  while  remaining  to  the  left  of  the 
four  red  poles  and  then  land  in  the  correct  LZ. 

4)  When  ready,  click  OK  (button  10)  to  start  the  timer. 

5)  Execute  the  Take-off  command. 

6)  Complete  one  lap  around  the  four  red  poles  and  stay  over  the  gray  path. 

7)  Land  on  the  (H)  building. 

8)  This  exercise  must  be  completed  in  3  minutes  50  seconds  (3:50)  or  less. 

Do  you  have  any  questions  about  the  practice  exercise? 


Experimenter: 

1 )  Run  the  mission  autonomously  with  waypoints  visible  and  point  out  the 
Landing  Zone  (LZ)  on  the  (H)  building. 

2)  Reload  the  properties  file  with  waypoints  visible. 

3)  Reload  the  mission  file. 

4)  Explain  that  the  operator  must  now  manually  pilot  the  MAV  around  the  gray 
pathway  while  remaining  to  the  left  of  the  4  red  poles  and  landing  in  the  correct 
LZ. 

5)  User  now  executes  the  Take-off  command. 

6)  User  completes  one  lap  around  the  red  poles  and  lands  on  the  (H)  building. 

7)  This  exercise  must  be  completed  in  3  minutes  50  seconds  (3:50)  or  less. 

8)  If  the  participant  crashes,  reload  the  properties  file  and  the  mission  file 
and  have  them  restart  the  exercise. 
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Practice  Exercise  6:  Load  the  correct  properties  file  for  each  condition  with  obstacle  course 
poles  (see  below) —  Then  load  the  mission  file:  obstaclecourse.ovm.  No  OneSAF  OTB 
scenario  is  required  for  this  mission. 

In  this  exercise  you  have  obstacles  to  navigate.  Complete  the  mission  by  flying  through  the 
series  of  red  and  green  poles,  and  then  return  to  your  start  point  to  take  two  snapshots  of  your 
C2  vehicle. 

We  will  go  over  the  practice  exercise  requirements  before  you  begin: 

1 )  The  experimenter  will  load  and  run  this  mission  autonomously.  Observe  how  the  MAV 
passes  to  the  right  of  all  green  poles  and  to  the  left  of  all  red  poles. 

2)  At  the  end  of  the  run  you  will  see  the  C2  vehicle  parked  on  the  road. 

3)  The  experimenter  will  now  reload  the  mission. 

4)  You  must  now  complete  the  course  with  a  few  additional  instructions:  After  you  finish 
navigating  around  the  poles,  you  will  need  to  take  snapshots  of  the  C2  vehicle 
with  both  cameras  before  landing  the  MAV. 

5)  When  ready,  click  OK  (button  10)  and  then  execute  Take-off. 

6)  Complete  the  obstacle  course. 

7)  Take  snapshot  of  C2  vehicle  from  camera  1 . 

8)  Take  snapshot  of  C2  vehicle  from  camera  2. 

9)  Land  near  the  C2  vehicle  -  but  do  NOT  land  on  the  C2  vehicle. 

10)  This  exercise  must  be  completed  in  5  minutes  (5:00)  or  less. 

Do  you  have  any  questions  about  the  practice  exercise? 


Experimenter: 

1)  Have  the  participant  complete  the  NASA  TLX  on  the  laptop  at  the  conclusion  of  practice 
exercise  6. 

2)  Then  give  the  paper-based  training  quiz  to  verify  that  the  operator  can  identify  all  of  the 
icons  and  control  buttons  used  in  the  study. 

3)  When  this  is  completed  you  may  proceed  to  the  mission  trials  on  the  next  page.  There  will 
be  five  total  missions  that  will  be  run  in  the  following  order: 

Mission  1  -  Racetrack  without  visible  waypoints  (Ft.  Polk). 

Mission  2  -  The  obstacle  course  with  snapshots  of  the  C2  vehicle  without  visible  waypoints  (Ft. 

Polk). 

Mission  3  -  The  multiple  target  ID/reconnaissance  mission  with  “unlimited”  time  (Ft.  Polk). 

Mission  4  -  T ransfer  skills  mission  without  visible  waypoints  (Ft.  McKenna). 

Mission  5  -Moving  target  reconnaissance  mission  with  “unlimited”  time  (Ft.  McKenna) 


Note:  Each  mission  will  have  a  corresponding  section  on  the  “observer  data  collection”  form 
where  the  experimenter  should  note  all  pertinent  information.  This  can  then  be  cross-referenced 
with  the  time  stamps  and  other  recorded  actions  contained  in  the  OCU  data  file. 
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Mission  Protocol 


There  will  be  5  missions  for  you  to  complete  during  this  portion  of  the  study.  The  experimenter 
will  instruct  you  on  mission  requirements  and  provide  any  documentation  necessary.  Once  you 
begin  a  mission,  the  experimenter  will  not  be  able  to  answer  questions  on  mission  requirements, 
so  please  ask  beforehand. 


Mission  1:  Racetrack  without  visible  waypoints  (Ft.  Polk). 

Stress  to  person  that  they  need  to  try  to  complete  this  mission  as  quickly  as  possible  but 

also  without  any  collisions  -if  they  have  a  collision,  restart  the  mission  immediately  and 

make  them  redo  until  they  complete  without  a  collision. 

with  racetrack  poles)-- 

cise. 

ow). 

mpletes  mission, 
ands  on  the  (H)  building. 

Experimenter: 

1 )  Load  the  correct  properties  file  for  each  condition  ( 

2)  No  OneSAF  OTB  scenario  is  required  for  this  exer 

3)  Read  over  the  mission  with  the  participant  (see  be 

4)  User  now  executes  the  Take-off  command  and  co 

5)  User  completes  one  lap  around  the  red  poles  and 

Mission  1: 


This  mission  is  a  repeat  of  practice  exercise  #5,  where  you  piloted  the  MAV  around  the  gray 
pathway  while  remaining  to  the  left  of  the  four  red  poles.  The  only  difference  is  that  there  will  be 
no  waypoints  on  the  satellite  map. 

You  need  to  try  to  complete  this  mission  as  quickly  as  possible  but  also  without  any  collisions  - 

if  you  have  a  collision,  you  will  be  required  to  redo  the  mission  from  the  beginning  until  you 

complete  it  without  a  collision. 

We  will  go  over  the  mission  requirements  before  you  begin: 

1 )  You  will  pilot  the  MAV  around  the  gray  pathway  while  remaining  to  the  left  of  the  four  red 
poles,  and  then  land  on  the  (H)  building. 

2)  When  ready,  press  OK  to  start  the  timer. 

3)  Execute  the  Take-off  command. 

4)  Complete  one  lap  around  the  four  red  poles  and  stay  over  the  gray  path. 

5)  Land  on  the  (H)  building. 

Do  you  have  any  questions  about  the  mission? 


Experimenter:  After  Mission  1,  have  the  participant  take  the  NASA  TLX  before  Mission  2. 
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Mission  2:  The  obstacle  course  with  snapshots  of  the  C2  vehicle  (Ft.  Polk). 

Stress  to  person  that  they  need  to  try  to  complete  this  mission  as  quickly  as  possible  but  also  without 

any  collisions  -if  they  have  a  collision,  restart  the  mission  immediately  and  make  them  redo  until  they 

complete  without  a  collision. 

Experimenter: 

1)  Load  the  correct  properties  file  with  obstacle  course  poles — 

2)  No  OneSAF  OTB  scenario  is  required  for  this  mission. 

3)  Read  over  the  mission  with  the  participant  (see  below). 

4)  Operator  should  now  take-off  and  complete  the  mission. 


Mission  2: 


This  mission  repeats  practice  exercise  #6,  where  you  navigated  a  series  of  red  and  green  poles. 
You  will  also  take  two  snapshots  of  the  C2  vehicle  at  the  end  of  the  run.  The  only  difference  is 
that  there  will  be  no  waypoints  on  the  satellite  map. 

You  need  to  try  to  complete  this  mission  as  quickly  as  possible  but  also  without  any  collisions  - 

if  you  have  a  collision,  you  will  be  required  to  redo  the  mission  from  the  beginning  until  you 

complete  it  without  a  collision. 

We  will  go  over  the  mission  requirements  before  you  begin: 

1 )  You  must  complete  the  obstacle  course. 

2)  After  you  finish  navigating  around  the  poles  (right  of  the  green  poles  and  left  of  the  red 
poles),  you  will  need  to  take  snapshots  of  the  C2  vehicle  with  both  cameras. 

3)  When  ready,  press  OK  and  then  execute  Take-off. 

4)  Complete  the  obstacle  course. 

5)  Take  snapshot  of  the  C2  vehicle  with  camera  1 . 

6)  Take  snapshot  of  the  C2  vehicle  with  camera  2. 

7)  Land,  but  do  NOT  land  on  the  C2  vehicle. 


Do  you  have  any  questions  about  the  mission? 


Experimenter:  After  Mission  2,  have  the  participant  take  the  NASA  TLX  before  Mission  3. 
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Mission  3:  The  multiple  target  ID/reconnaissance  mission  (Ft.  Polk). 

Experimenter: 

1 )  Load  the  OneSAF  scenario  Exp3Polk6targets.2.gz 

2)  Load  the  correct  properties  file: 

3)  No  mission  file  is  required  for  this  mission. 

4)  Advise  the  participant  to  click  OK  to  begin  the  mission  and  start  looking  for  entities  to 
identify  via  the  camera  snapshots. 

5)  Make  notes  on  the  data  collection  form  as  to  any  entities  that  are  identified  more  than 
once,  as  well  asking  any  mission  related  questions  as  outlined  on  the  same  form. 

6)  The  user  should  land  when  all  targets  have  been  identified  or  when  the  15  minute  mark  is 
reached. 


Mission  3: 


This  mission  involves  using  the  MAV  to  do  reconnaissance  work.  This  is  primarily  a  target 
identification  mission.  You  need  to  complete  this  mission  as  quickly  as  possible. 

We  will  go  over  the  mission  requirements  before  you  begin: 

1)  You  need  to  find  and  identify  (take  snapshots)  all  6  targets  on  the  map  (see  Intel  sheet 
on  the  next  page),  with  a  maximum  search  time  of  15  minutes. 

2)  Positive  ID  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  both  the 
forward  and  downward  cameras,  and  each  entity  must  be  centered  in  the  frame  so  that 
the  center  ( )  overlay  is  touching  part  of  the  entity.  The  entity  doesn’t  need  to  be  perfectly 
centered  in  the  overlay. 

3)  When  ready  press  OK  to  begin  the  mission  and  start  the  timer. 

4)  Begin  looking  for  entities  to  identify  via  the  cameras. 

5)  Land  the  MAV  immediately  after  finding  and  photographing  all  targets. 

Do  you  have  any  questions  about  the  mission? 


Experimenter:  After  completing  Mission  3,  have  the  participant  take  the  NASA  TLX  before 
Mission  4.  Inform  the  participant  that  the  next  two  missions  will  be  in  new  simulated 
environments. 
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Mission  3:  Intel  Sheet 


The  entities  shown  below  have  been  identified  as  having  terrorist  involvement  within  the  military 
base.  Your  mission  will  be  to  scout  the  base  area  and  identify  all  6  of  these  enemy  targets. 

Positive  identification  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  BOTH  the 
forward  and  downward  cameras,  and  each  entity  must  be  centered  in  the  frame  so  that  the 
center  framing  (  )  overlay  is  touching  the  entity  in  the  picture.  The  entity  doesn’t  need  to  be 
perfectly  centered  in  the  overlay.  Find  and  photograph  all  entities  as  quickly  as  you  can.  The 
mission  will  end  if  targets  have  not  been  found  within  15  minutes. 

DO  NOT  crash  the  MAV  vehicle!  Land  the  MAV  immediately  after  finding  and  photographing  all 
targets. 


(A)  2  Tanks  with  Armed  Missiles 
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Mission  4:  The  transfer  skills  obstacle  course  (Ft.  McKenna). 

Stress  to  person  that  they  need  to  try  to  complete  this  mission  as  quickly  as  possible  but  also  without 

any  collisions  -if  they  have  a  collision,  restart  the  mission  immediately  and  make  them  redo  until  they 

complete  without  a  collision. 

Experimenter: 

1 )  Load  the  correct  properties  file  with  new  environment  and  obstacle  poles 

2)  Load  the  mission  file:  McKenna  Obstacle  Course. ovm 

3)  No  OneSAF  OTB  scenario  is  required  for  this  exercise. 

4)  Read  the  mission  with  the  participant  (see  below). 

5)  Run  the  mission  autonomously  with  waypoints  visible. 

6)  Inform  the  participants  that  they  must  complete  the  course  (without  crashing)  in  as  many  tries 
as  necessary  before  moving  on  the  Mission  5 — there  is  no  time  limit  for  this  particular  mission. 

7)  Reload  the  properties  file. 

8)  User  now  executes  the  Take-off  command  and  completes  mission. 


Mission  4: 


The  final  two  missions  will  be  in  a  new  simulated  environment.  Mission  4  involves  using  the 
MAV  to  navigate  around  obstacles.  Complete  the  mission  by  piloting  the  MAV  around  the 
roadway,  flying  through  the  series  of  red  and  green  poles,  and  then  landing  in  the  designated 
area.  You  must  fly  to  the  left  of  the  red  poles  and  to  the  right  of  the  green  poles. 

You  need  to  try  to  complete  this  mission  as  quickly  as  possible  but  also  without  any  collisions  - 

if  you  have  a  collision,  you  will  be  required  to  redo  the  mission  from  the  beginning  until  you 

complete  it  without  a  collision.  We  will  go  over  the  mission  requirements  before  you  begin: 

1 )  The  experimenter  will  load  and  run  this  mission  autonomously  with  waypoints  visible. 
Observe  how  the  MAV  passes  to  the  right  of  all  green  poles  and  to  the  left  of  all  red 
poles. 

2)  You  must  now  complete  the  course.  You  must  navigate  the  obstacle  course  without 
crashing  before  you  can  move  on  to  the  final  mission. 

3)  When  ready,  press  OK  and  then  execute  Take-off. 

4)  Complete  the  obstacle  course. 

5)  Land  in  the  designated  area  beside  the  church. 

6)  This  exercise  must  be  completed  without  crashing  in  as  many  tries  as  necessary.  There 
is  no  time  limit. 

Do  you  have  any  questions  about  the  mission? 


Experimenter:  After  completing  Mission  4,  have  the  participant  take  the  NASA  TLX 
before  Mission  5. 
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Mission  5:  The  multiple  target  ID/reconnaissance  mission  (Ft.  McKenna). 

Experimenter: 

1)  Load  the  OneSAF  OTB  scenario  Exp3McKenna6targets.1  .gz 

2)  Load  the  correct  properties  file — 

3)  No  mission  file  is  required  for  this  mission. 

4)  Make  notes  on  the  data  collection  form  as  to  any  entities  that  are  identified  more  than 
once,  as  well  asking  any  mission  related  questions  as  outlined  on  the  same  form. 

5)  The  user  does  not  need  to  land  unless  all  targets  have  been  identified.  When  the  timer 
reaches  15  minutes  and  the  operator  hasn’t  found  all  targets,  ask  him  or  her  to  land. 


Mission  5 


This  mission  involves  using  the  MAV  to  do  more  reconnaissance  work.  This  is  primarily  a  target 
identification  mission.  You  need  to  complete  this  mission  as  quickly  as  possible. 

We  will  go  over  the  mission  requirements  before  you  begin: 

1 )  The  experimenter  will  load  the  mission  files  and  scenario. 

2)  You  need  to  find  and  identify  (take  snapshots)  all  6  targets  on  the  map  (see  Intel  sheet 
on  next  page),  with  a  maximum  search  time  of  15  minutes. 

3)  Positive  ID  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  both  the 
forward  and  downward  cameras,  and  each  entity  must  be  centered  in  the  frame  so  that 
the  center  ( )  overlay  is  touching  part  of  the  entity.  The  entity  doesn’t  need  to  be  perfectly 
centered  in  the  overlay. 

4)  When  ready  press  OK  to  begin  the  mission  and  start  the  timer. 

5)  Begin  looking  for  entities  to  identify  via  the  cameras. 

6)  Land  the  MAV  immediately  after  finding  and  photographing  all  targets. 


Do  you  have  any  questions  about  the  mission? 


After  Mission  5,  complete  the  computer-based  questionnaire;  then  proceed  to  the 
final  debriefing  session. 
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Mission  5:  Intel  Sheet 


The  entities  shown  below  have  been  identified  as  having  terrorist  involvement  within  the  military 
base.  Your  mission  will  be  to  scout  the  base  area  and  identify  all  6  of  these  enemy  targets. 

Positive  identification  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  BOTH  the 
forward  and  downward  cameras,  and  each  entity  must  be  centered  in  the  frame  so  that  the 
center  framing  (  )  overlay  is  touching  the  entity  in  the  picture.  The  entity  doesn’t  need  to  be 
perfectly  centered  in  the  overlay.  Find  and  photograph  all  entities  as  quickly  as  you  can.  The 
mission  will  end  if  targets  have  not  been  found  within  15  minutes. 

DO  NOT  crash  the  MAV  vehicle!  Land  the  MAV  immediately  after  finding  and  photographing  all 
targets. 


(D)  ^  Armed  Soldier 


D-25 


Experimenter:  After  Mission  5  is  completed... 

1)  Have  the  participant  take  the  NASA  TLX 

2)  Give  participant  the  Debrief  sheet,  and  field  any  questions  he/she  may  have 

3)  If  no  questions,  proceed  to  either  give  participants  monetary  compensation  or 
point  compensation. 

4)  Before  they  leave,  give  them  the  Psychology  Participant  Research  Experience 
Form. 
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